Re: [openstack-dev] Paste unmaintained

[Thomas Herve]

On Thu, Aug 2, 2018 at 4:27 PM, Doug Hellmann  wrote:
> Excerpts from Stephen Finucane's message of 2018-08-02 15:11:25 +0100:
>> tl;dr: It seems Paste [1] may be entering unmaintained territory and we
>> may need to do something about it.
>>
>> I was cleaning up some warning messages that nova was issuing this
>> morning and noticed a few coming from Paste. I was going to draft a PR
>> to fix this, but a quick browse through the Bitbucket project [2]
>> suggests there has been little to no activity on that for well over a
>> year. One particular open PR - "Python 3.7 support" - is particularly
>> concerning, given the recent mailing list threads on the matter.
>>
>> Given that multiple projects are using this, we may want to think about
>> reaching out to the author and seeing if there's anything we can do to
>> at least keep this maintained going forward. I've talked to cdent about
>> this already but if anyone else has ideas, please let me know.
>>
>> Stephen
>>
>> [1] https://pypi.org/project/Paste/
>> [2] https://bitbucket.org/ianb/paste/
>> [3] https://bitbucket.org/ianb/paste/pull-requests/41
>>
>
> The last I heard, a few years ago Ian moved away from Python to
> JavaScript as part of his work at Mozilla. The support around
> paste.deploy has been sporadic since then, and was one of the reasons
> we discussed a goal of dropping paste.ini as a configuration file.
>
> Do we have a real sense of how many of the projects below, which
> list Paste in requirements.txt, actually use it directly or rely
> on it for configuration?
>
> Doug
>
> $ beagle search --ignore-case --file requirements.txt 'paste[><=! ]'
> +++--++
> | Repository | Filename   
> | Line | Text   |
> +++--++
> | airship-armada | requirements.txt   
> |8 | Paste>=2.0.3   |
> | airship-deckhand   | requirements.txt   
> |   12 | Paste # MIT|
> | anchor | requirements.txt   
> |9 | Paste # MIT|
> | apmec  | requirements.txt   
> |6 | Paste>=2.0.2 # MIT |
> | barbican   | requirements.txt   
> |   22 | Paste>=2.0.2 # MIT |
> | cinder | requirements.txt   
> |   37 | Paste>=2.0.2 # MIT |
> | congress   | requirements.txt   
> |   11 | Paste>=2.0.2 # MIT |
> | designate  | requirements.txt   
> |   25 | Paste>=2.0.2 # MIT |
> | ec2-api| requirements.txt   
> |   20 | Paste # MIT|
> | freezer-api| requirements.txt   
> |8 | Paste>=2.0.2 # MIT |
> | gce-api| requirements.txt   
> |   16 | Paste>=2.0.2 # MIT |
> | glance | requirements.txt   
> |   31 | Paste>=2.0.2 # MIT |
> | glare  | requirements.txt   
> |   29 | Paste>=2.0.2 # MIT |
> | karbor | requirements.txt   
> |   28 | Paste>=2.0.2 # MIT |
> | kingbird   | requirements.txt   
> |7 | Paste>=2.0.2 # MIT |
> | manila | requirements.txt   
> |   30 | Paste>=2.0.2 # MIT |
> | meteos | requirements.txt   
> |   29 | Paste # MIT|
> | monasca-events-api | requirements.txt   
> |6 | Paste # MIT|
> | monasca-log-api| requirements.txt   
> |6 | Paste>=2.0.2 # MIT |
> | murano | requirements.txt   
> |   28 | Paste>=2.0.2 # MIT |
> | neutron| requirements.txt   
> |6 | Paste>=2.0.2 # MIT |
> | nova   | requirements.txt   
> |   19 | Paste>=2.0.2 # MIT |
> | novajoin  

Re: [openstack-dev] [keystone][monasca][congress][senlin][telemetry] authenticated webhook notifications

[Thomas Herve]

On Sat, May 5, 2018 at 1:53 AM, Eric K  wrote:
> Thanks a lot Witold and Thomas!
>
> So it doesn't seem that someone is currently using a keystone token to
> authenticate web hook? Is is simply because most of the use cases had
> involved services which do not use keystone?
>
> Or is it unsuitable for another reason?

It's fairly impractical for webhooks because

1) Tokens expire fairly quickly.
2) You can't store all the data in the URL, so you need to store the
token and the URL separately.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][monasca][congress][senlin][telemetry] authenticated webhook notifications

[Thomas Herve]

On Thu, May 3, 2018 at 9:49 PM, Eric K  wrote:
> Question to the projects which send or consume webhook notifications
> (telemetry, monasca, senlin, vitrage, etc.), what are your
> supported/preferred authentication mechanisms? Bearer token (e.g.
> Keystone)? Signing?
>
> Any pointers to past discussions on the topic? My interest here is having
> Congress consume and send webhook notifications.
>
> I know some people are working on adding the keystone auth option to
> Monasca's webhook framework. If there is a project that already does it,
> it could be a very helpful reference.

Hi,

I'll add a few that you didn't mention which consume such webhooks.

 * Heat has been using EC2 signatures basically since forever. It
creates EC2 credentials for a Keystone user, and signs URL that way.
 * Zaqar has signed URLs
(https://developer.openstack.org/api-ref/message/#pre-signed-queue)
which allows sharing queues without authentication.
 * Swift temp URLs
(https://docs.openstack.org/swift/latest/middleware.html#tempurl) is a
good mechanism to share information as well.

I'd say application credentials would make those operations a bit
nicer, but they are not completely there yet. Everybody not
reinventing its own wheel would be nice too :).

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][TripleO] - Getting attributes of openstack resources not created by the stack for TripleO NetworkConfig.

[Thomas Herve]

On Mon, Apr 23, 2018 at 8:09 PM, Dan Sneddon  wrote:

> We could add the ControlPlaneDefaultRoute and ControlPlaneSubnetCidr to
> network_data.yaml, but this would involve some duplication of configuration
> data, since those are currently defined in undercloud.conf. A more robust
> solution might be to generate network_data.yaml from that info in
> undercloud.conf, but currently we don't modify any files in the
> tripleo-heat-templates package after it gets installed.

Right, it seems getting those values from Neutron is better.

> I can't speak to the roadmap of Heat/Neutron/Nova on the undercloud, for the
> immediate future I don't see us moving away from Heat entirely due to
> upgrade requirements.
>
> I can see another use case for this Heat functionality, which is that I
> would like to be able to generate a report using Heat that lists all the
> ports in use in the entire deployment. This would be generated
> post-deployment, and could be used to populate an external DNS server, or
> simply to report on which IPs belong to which nodes.

Jiri wrote a small tool that does mostly that:
https://gist.github.com/jistr/ad385d77db7600c18e8d52652358b616
We could make that more official, but we already have the info.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][TripleO] - Getting attributes of openstack resources not created by the stack for TripleO NetworkConfig.

[Thomas Herve]

On Mon, Apr 23, 2018 at 7:16 PM, Harald Jensås <hjen...@redhat.com> wrote:
> On Fri, 2018-04-20 at 14:44 +0200, Thomas Herve wrote:

>> To make it clear, what you want to prevent is the need to add more
>> keys in network_data.yaml?
>>
>> As those had to be provided at some point, I wonder if tripleo can't
>> find a way to pass them again on the overcloud deploy.
>>
> No, the networks defined in network_data.yaml is fine, that is the data
> used to create the neutron stuff so passing the data from there is
> already in place to some extent.
>
> But, the ctlplane network is not defined in network_data.yaml.

OK.
>> If you can list the attributes you care about that'd be great.
>>
>
> Guess what I envision is a client_config attribute, a map with data
> useful to configure a network interface on the client. (I put * on the
> ones I believe could be useful for TripleO)
>
> * /v2.0/networks/{network_id}/mtu
> /v2.0/networks/{network_id}/dns_domain
> * /v2.0/subnets/{subnet_id}/dns_nameservers
> * /v2.0/subnets/{subnet_id}/host_routes
> /v2.0/subnets/{subnet_id}/ip_version
> * /v2.0/subnets/{subnet_id}/gateway_ip
> * /v2.0/subnets/{subnet_id}/cidr
> * /v2.0/subnets/{subnet_id}/ipv6_address_mode
> * /v2.0/subnets/{subnet_id}/ipv6_ra_mode
> /v2.0/ports/{port_id}/description - Why not?
> /v2.0/ports/{port_id}/dns_assignment
> /v2.0/ports/{port_id}/dns_domain
> /v2.0/ports/{port_id}/dns_name
> * /v2.0/ports/{port_id}/fixed_ips - We have this already
> /v2.0/ports/{port_id}/name- Why not?

I think we have most of those on resources already. From the required
ones, I think the only ones mising are ipv6_address_mode and
ipv6_ra_mode on subnets. If we make external resources work, it'll be
easy to provide what you need.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][TripleO] - Getting attributes of openstack resources not created by the stack for TripleO NetworkConfig.

[Thomas Herve]

On Thu, Apr 19, 2018 at 2:59 PM, Harald Jensås  wrote:
> Hi,

Hi, thanks for sending this. Responses inline.

> When configuring TripleO deployments with nodes on routed ctlplane
> networks we need to pass some per-network properties to the
> NetworkConfig resource[1] in THT. We get the ``ControlPlaneIp``
> property using get_attr, but the NIC configs need a couple of more
> parameters[2], for example: ``ControlPlaneSubnetCidr``,
> ``ControlPlaneDefaultRoute`` and ``DnsServers``.
>
> Since queens these templates are jinja templated, to generate things
> from from network_data.yaml. When using routed ctlplane networks, the
> parameters ``ControlPlaneSubnetCidr`` and ``ControlPlaneDefaultRoute``
> will be different. So we need to use static per-role
> Net::SoftwareConfig templates, and add parameters such as
> ``ControlPlaneDefaultRouteLeafX``.
>
> The values the use need to pass in for these are already available in
> the neutron ctlplane network configuration on the undercloud. So
> ideally we should not need to ask the user to provide them in
> parameter_defaults, we should resolve the correct values automatically.

To make it clear, what you want to prevent is the need to add more
keys in network_data.yaml?

As those had to be provided at some point, I wonder if tripleo can't
find a way to pass them again on the overcloud deploy.

Inspecting neutron is an elegant solution, though.

> : We can get the port ID using get_attr:
>
>  {get_attr: [, addresses, , 0, port]}
>
> : From there outside of heat we can get the subnet_id:
>
>  openstack port show 2fb4baf9-45b0-48cb-8249-c09a535b9eda \
>  -f yaml -c fixed_ips
>
>  fixed_ips: ip_address='172.20.0.10', subnet_id='2b06ae2e-423f-4a73-
> 97ad-4e9822d201e5'
>
> : And finally we can get the gateway_ip and cidr of the subnet:
>
>   openstack subnet show 2b06ae2e-423f-4a73-97ad-4e9822d201e5 \
>   -f yaml -c gateway_ip -c cidr
>
>  cidr: 172.20.0.0/26
>  gateway_ip: 172.20.0.62
>
>
> The problem is getting there using heat ...
> a couple of ideas:
>
> a) Use heat's ``external_resource`` to create a port resource,
>and then  a external subnet resource. Then get the data
>from the external resources. We probably would have to make
>it possible for a ``external_resource`` depend on the server
>resource, and verify that these resource have the required
>attributes.

I believe that's a relatively easy fix. It's unclear why we didn't
allow that in the first place, probably because we were missing a use
case, but it seems valuable here.

> b) Extend attributes of OS::Nova::Server (OS::Neutron::Port as
>well probably) to include the data.
>
>If we do this we should probably aim to be in parity with
>what is made available to clients getting the configuration
>from dhcp. (mtu, dns_domain, dns_servers, prefixlen,
>gateway_ip, host_routes, ipv6_address_mode, ipv6_ra_mode
>etc.)

I'm with you on exposing more neutron data to the Port resource. It
can be complicated because some of them are implementation specific,
but we can look into those.

I don't think adding them directly to the Server resource makes a ton
of sense though.

> c) Create a new heat function to read properties of any
>openstack resource, without having to make use of the
>external_resource in heat.

It's an interesting idea, but I think it would look a lot like what
external resources are supposed to be.

I see a few changes:
 * Allow external resource to depend on other resources
 * Expose more port attributes
 * Expose more subnet attributes

If you can list the attributes you care about that'd be great.

Thanks,

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Nominating yangzhenyu for Zaqar core

[Thomas Herve]

On Mon, Feb 26, 2018 at 2:38 AM, Fei Long Wang  wrote:
> Hi team,
>
> I would like to propose adding Zhenyu Yang(yangzhenyu) for the Zaqar core 
> team. He has been an awesome contributor since joining the Zaqar team. And 
> now he is the most active non-core contributor on Zaqar projects for the last 
> 180 days[1]. Zhenyu has great technical expertise and contributed many high 
> quality patches. I'm sure he would be an excellent addition to the team. If 
> no one objects, I'll proceed and add him in a week from now. Thanks.
>
> [1] http://stackalytics.com/report/contribution/zaqar-group/180

+1!

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] cyclomatic complexity check in flake8 & Python 3

[Thomas Herve]

On Tue, Dec 12, 2017 at 2:08 PM, Thierry Carrez  wrote:
> Zane Bitter wrote:
>> We have around 60 repos using the 'mccabe' package to do cyclomatic
>> complexity checks in flake8 in their gate jobs.[1] Based on advice
>> received at the time[2], the version of mccabe that we use in the gate
>> is effectively pinned to 0.2.1 by hacking (the project, not... never
>> mind).[3]
>>
>> This is starting to cause problems, because version 0.2.1 does not work
>> with Python 3. By 'does not work' I don't mean that it refuses to run, I
>> mean that it returns completely different numbers. (As far as I can tell
>> they're always the same or lower.) tox creates the pep8 environment
>> using the version of python that tox itself is running in, so as distros
>> increasingly prefer to install packages using python3, people will
>> increasingly be running flake8 under python3, and they'll run into
>> situations like the one I had today where their pep8 tests pass locally
>> but fail in the gate (which still runs under python2). When the gate
>> switches to python3, as I assume it eventually must, we'll get bogus
>> results.
>>
>> Unfortunately, moving to a later version is complicated by the fact that
>> 0.2.1 was... not good software. There is exactly one regression test.[4]
>> And if you're about to ask whether it was to check the trivial case that
>> the function:
>>
>>   def foo():
>>   return None
>>
>> has a cyclomatic complexity of 1, then the joke is on you because it
>> actually returns 2 for that function.
>>
>> Later versions have more tests and appear to be consistent with each
>> other and across python versions, but they return completely different
>> results to 0.2.1. And the results are mostly higher, so if we bump the
>> requirements many of those 60 projects will break.
>>
>> (I could also go into details on how the numbers that later versions
>> report are also wrong in different ways, but I'll save it for another day.)
>>
>> Here's how the allegedly most complex function in Heat[5] shakes out,
>> for example:
>>
>>   mccabe  py27  py36
>>   0.2.114 6
>>   0.3.12323
>>   0.6.12323
>>
>> I don't have a solution to suggest, but I burned most of my day digging
>> into this so I just wanted to let y'all know how screwed we are.
> Thanks for digging into this, Zane. I'd say we have two options, based
> on how useful running those tests is:
>
> 1/ If they are useful, we should bump the version, at the same time
> adjusting max-complexity per repo to match the most complex function
> (with some slack factor). Encourage more projects to use cyclomatic
> complexity checks and fix the biggest offenders using cycle goals.
>
> 2/ If they are not useful, just drop cyclomatic complexity tests overall
> rather than relying on wrong results and wasting CPU cycles

Hugely in favor of doing that. Even if it was working properly, I
don't think for Heat it was ever a useful test. We end up either
bumping the max or moving a conditional in a function, which doesn't
solve anything.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Nominating gecong for Zaqar core

[Thomas Herve]

On Tue, Oct 10, 2017 at 11:15 PM, feilong  wrote:
> Hi team,
>
> I would like to propose adding Cong Ge(gecong) for the Zaqar core team.
> He has been an awesome contributor since joining the Zaqar team. And now
> he is currently the most active non-core reviewer on Zaqar projects for
> the last 180 days[1]. Cong has great technical expertise and contributed
> many high quality patches. I'm sure he would be an excellent addition to
> the team. If no one objects, I'll proceed and add him in a week from
> now. Thanks.

+1!

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] revised structure of the heat-templates repository. Suggestions

[Thomas Herve]

On Fri, May 19, 2017 at 5:00 PM, Lance Haig  wrote:
> Hi,

Hi Lance,

Thanks for starting this. Comments inline.

> As we know the heat-templates repository has become out of date in some
> respects and also has been difficult to be maintained from a community
> perspective.

While it has been out of date, I'm not sure it's because it's been
difficult. We just don't have the manpower or didn't dedicate enough
time to it.

> For me the repository is quiet confusing with different styles that are used
> to show certain aspects and other styles for older template examples.
>
> This I think leads to confusion and perhaps many people who give up on heat
> as a resource as things are not that clear.
>
> From discussions in other threads and on the IRC channel I have seen that
> there is a need to change things a bit.
>
>
> This is why I would like to start the discussion that we rethink the
> template example repository.
>
> I would like to open the discussion with mys suggestions.
>
> We need to differentiate templates that work on earlier versions of heat
> that what is the current supported versions.
>
> I have suggested that we create directories that relate to different
> versions so that you can create a stable version of examples for the heat
> version and they should always remain stable for that version and once it
> goes out of support can remain there.
> This would mean people can find their version of heat and know these
> templates all work on their version

So, a couple of things:
* Templates have a version field. This clearly shows on which version
that template ought to work.
* Except when some resources changed (neutron loadbalancer, some
ceilometer stuff), old templates should still work. If they don't,
it's a bug. Obviously we won't fix it on unmaintained versions, but we
work really hard at maintaining compatibility. I'd be surprised to
find templates that are really broken.

It'd probably be nice to update all templates to the latest supported
version. But we don't remove old versions of templates, so it's also
good to keep them around, if updating the versions doesn't bring
anything new.

> We should consider adding a docs section that that includes training for new
> users.
>
> I know that there are documents hosted in the developer area and these could
> be utilized but I would think having a documentation section in the
> repository would be a good way to keep the examples and the documents in the
> same place.
> This docs directory could also host some training for new users and old ones
> on new features etc.. In a similar line to what is here in this repo
> https://github.com/heat-extras/heat-tutorial

I'd rather see documentation in the main repository. It's nice to have
some stuff in heat-templates, but there is little point if the doc
isn't published anywhere. Maybe we could have links?

> We should include examples form the default hooks e.g. ansible salt etc...
> with SoftwareDeployments.
>
> We found this quiet helpful for new users to understand what is possible.

We have those AFAIU:
https://github.com/openstack/heat-templates/tree/master/hot/software-config/example-templates

> We should make sure that the validation running against the templates runs
> without ignoring errors.
>
> This was noted in IRC that some errors were ignored as the endpoints or
> catalog was not available. It would be good to have some form of headless
> catalog server that tests can be run against so that developers of templates
> can validate before submitting patches.

Yep that's a good idea, I've written a patch here:
https://review.openstack.org/465860

Thanks,

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Heat] Conditionally passing properties in Heat

[Thomas Herve]

On Wed, Apr 12, 2017 at 9:00 PM, Dan Sneddon  wrote:
> I'm implementing predictable control plane IPs for spine/leaf, and I'm
> running into a problem implementing this in the TripleO Heat templates.
>
> I have a review in progress [1] that works, but fails on upgrade, so I'm
> looking for an alternative approach. I'm trying to influence the IP
> address that is selected for overcloud nodes' Control Plane IP. Here is
> the current construct:
>
>   Controller:
> type: OS::TripleO::Server
> metadata:
>   os-collect-config:
> command: {get_param: ConfigCommand}
> properties:
>   image: {get_param: controllerImage}
>   image_update_policy: {get_param: ImageUpdatePolicy}
>   flavor: {get_param: OvercloudControlFlavor}
>   key_name: {get_param: KeyName}
>   networks:
> - network: ctlplane  # <- Here's where the port is created
>
> If I add fixed_ip: to the networks element at the end of the above, I
> can select an IP address from the 'ctlplane' network, like this:
>
>   networks:
> - network: ctlplane
>   fixed_ip: {get_attr: [ControlPlanePort, ip_address]}
>
> But the problem is that if I pass a blank string to fixed_ip, I get an
> error on deployment. This means that the old behavior of automatically
> selecting an IP doesn't work.
>
> I thought I has solved this by passing an external Neutron port, like this:
>
>   networks:
> - network: ctlplane
>   port: {get_attr: [ControlPlanePort, port_id]}
>
> Which works for deployments, but that fails on upgrades, since the
> original port was created as part of the Nova::Server resource, instead
> of being an external resource.

Can you detail how it fails? I was under the impression we never
replaced servers no matter what (or we try to do that, at least). Is
the issue that your new port is not the correct one?

> I'm now looking for a way to use Heat conditionals to apply the fixed_ip
> only if the value is not unset. Looking at the intrinsic functions [2],
> I don't see a way to do this. Is what I'm trying to do with Heat possible?

You should be able to write something like that (not tested):

networks:
  if:
- 
- network: ctlplane
  fixed_ip: {get_attr: [ControlPlanePort, ip_address]}
- network: ctlplane

The question is how to define your condition. Maybe:

conditions:
  fixed_ip_condition:
 not:
equals:
  - {get_attr: [ControlPlanePort, ip_address]}
  - ''

To get back to the problem you stated first.


> Another option I'm exploring is conditionally applying resources. It
> appears that would require duplicating the entire TripleO::Server stanza
> in *-role.yaml so that there is one that uses fixed_ip and one that does
> not. Which one is applied would be based on a condition that tested
> whether fixed_ip was blank or not. The downside of that is that it would
> make the role definition confusing because there would be a large
> resource that was implemented twice, with only one line difference
> between them.

You can define properties with conditions, so you shouldn't need to
rewrite everything.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][ui] [heat] i18n proposal for heat templates 'description' help strings

[Thomas Herve]

On Thu, Mar 23, 2017 at 9:39 AM, Peng Wu  wrote:
> Hi,
>
>   For TripleO UI project, some users requested to translate the web UI.
> But some web UI string are from heat template files in tripleo-heat-
> templates project.
>
>   In order to get translated templates displayed in tripleo-ui, we
> propose a blueprint as follows, which needs to change code in heat,
> tripleo-heat-tempates and tripleo-ui projects.
>
>   I18n proposal for heat templates 'description' help strings
>
>   1. Update heat project to accept "translation-domain" header in
> RESTful request, like "translation-domain: tripleo-heat-templates"
>
>  a. With "translation-domain" header, heat will try to translate
> "title" and "description" field using "tripleo-heat-templates.po"
>
>  b. Without "translation-domain" header, heat will return the
> fields like before
>
>  c. Add some field in config file for security to have a list of
> allowed "translation-domain",
> like "allowed-translation-domains: ['tripleo-heat-templates',
> ...]"

>From the Heat side of things, that sounds like a big no-no to me.
While we've done many things to cater to TripleO, this is way too
specific of a use case. It doesn't even make sense for the general use
case of passing user templates to Heat.


>   2. Update tripleo-heat-templates to generate the translation files,
>  like "tripleo-heat-templates.pot"
>
>  a. May need to write python script to extract "title" and
> "description" field from yaml files
>
>  b. May need to integrate into python babel config or use separate
> po files
>
>
>   3. Update tripleo-ui to use locale API with "translation-domain"
> header to ask the RESTful response with translated "title" and
> "description" fields from heat services
>
>  a. tripleo-ui will send request with two additional headers:
> "Accept-Language" and "translation-domain: tripleo-heat-
> templates"

Those last 2 steps may make more sense. Possibly try to store those
translation files somewhere and manage them in the UI?

Thanks,

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Thomas Herve]

On Wed, Mar 22, 2017 at 3:24 PM, Alex Schultz  wrote:
> On Wed, Mar 22, 2017 at 7:58 AM, Paul Belanger  wrote:
[snip]
>> Please correct me if I am wrong, because I still have my container training
>> wheels on. I understand the need for etcd, and operators to write their
>> configuration into it.  Why I am struggling with still, is why you need
>> oslo.config to support it.  There is nothing stopping an operator today from
>> using etcd / confd in a container, right?  I can only imagine countless other
>> services that run in containers using them.
>>
>
> We want oslo.config to support it as a source for configuration.
> Dealing with files in containers is complicated. If we can remove the
> requirement to munge configurations for containers,
> deployment/updating containers becomes easier.  The service container
> becomes a single artifact to be deployed with less moving parts which
> helps reduce complexity and errors.  The process for moving a single
> container artifact is a lot easier than moving container and updating
> configurations based on where it's landing.

I believe the point is that operators will need to have a solution for
non-oslo services, if we want to centralize configuration using etcd.
If that's the case, it's unclear what will be the benefit of having
direct support for etcd in oslo.

I have even a counter example: let's say you want to deploy heat api
using httpd (as recommended). You'll deploy it it in a container: you
then need confd to manage httpd config, but Heat would then talk
directly to etcd? I'm not sure the benefit would be gigantic.

To summarize: confd (or something equivalent) needs to be in the
equation. Should we "simply" standardize on it?

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Thomas Herve]

On Tue, Mar 21, 2017 at 10:26 PM, Davanum Srinivas  wrote:
> Jay,
>
> the /v3alpha HTTP API  (grpc-gateway) supports watch
> https://coreos.com/etcd/docs/latest/dev-guide/apispec/swagger/rpc.swagger.json

Ah, that's a great find, thanks. That means that we don't have to use
grpc, and can still talk to an HTTP endpoint that would integrate
better with our tools. I tested the API, and it works fine.

Regarding reloading, I don't think watch is mandatory for now. As a
first step, I would fetch the keys again upload reload, like we do for
files.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Thomas Herve]

On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow  wrote:

> * How does reloading work (does it)?

No. There is nothing that we can do in oslo that will make services
magically reload configuration. It's also unclear to me if that's
something to do. In a containerized environment, wouldn't it be
simpler to deploy new services? Otherwise, supporting signal based
reload as we do today should be trivial.

> * What's the operational experience (editing a ini file is about the lowest
> bar we can possible get to, for better and/or worse).
>
> * Does this need to be a new oslo.config backend or is it better suited by
> something like the following (external programs loop)::
>
>etcd_client = make_etcd_client(args)
>while True:
>has_changed = etcd_client.get_new_config("/blahblah") # or use a
> watch
>if has_changed:
>   fetch_and_write_ini_file(etcd_client)
>   trigger_reload()
>time.sleep(args.wait)

That's confd: https://github.com/kelseyhightower/confd/ . Bonus
points; it supports a ton of other backends. One solution is to
provide templates and documentation to use confd with OpenStack.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral] Mistral Custom Actions API Design

[Thomas Herve]

On Fri, Mar 10, 2017 at 9:52 PM, Ryan Brady  wrote:
>
> One of the pain points for me as an action developer is the OpenStack
> actions[1].  Since they all use the same method name to retrieve the
> underlying client, you cannot simply inherit from more than one so you are
> forced to rewrite the client access methods.  We saw this in creating
> actions for TripleO[2].  In the base action in TripleO, we have actions that
> make calls to more than one OpenStack client and so we end up re-writing and
> maintaining code.  IMO the idea of using multiple inheritance there would be
> helpful.  It may not require the mixin approach here, but rather a design
> change in the generator to ensure the method names don't match.

Is there any reason why those methods aren't functions? AFAICT they
don't use the instance, they could live top level in the action module
and be accessible by all actions. If you can avoid multiple
inheritance (or inheritance!) you'll simplify the design. You could
also do client = NovaAction().get_client() in your own action (if
get_client was a public method).

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] pycrypto is dead, long live pycryptodome... or cryptography...

[Thomas Herve]

On Wed, Mar 8, 2017 at 8:03 PM, Matthew Thode  wrote:
> So, pycrypto upstream is dead and has been for a while, we should look
> at moving off of it for both bugfix and security reasons.
>
> Currently it's used by the following.
>
> barbican, cinder, trove, glance, heat, keystoneauth, keystonemiddleware,
> kolla, openstack-ansible, and a couple of other smaller places.

Heat keeps it mostly for (old) backward compatibility. We can possibly
remove it now, especially if it helps global requirements.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [telemetry][heat] Removal of Aodh combination alarms

[Thomas Herve]

On Tue, Feb 28, 2017 at 7:54 AM, Sergey Kraynev  wrote:
> Zane,
> as I understand, all related code was removed. Probably we may to remove our
> Hidden resource too.

We can't remove our hidden resources, because it breaks stacks that
reference them (to the point that you can't delete them).

> If we want to have hidden resource, then probably as you mentioned will be
> enough to raise Exceptions and execute nothing for handle_delete.

We don't want to, we have to :). One trick I'd like to check is simply
have an alias to Heat::None. It may be good enough.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable][heat] Heat stable-maint additions

[Thomas Herve]

On Fri, Feb 17, 2017 at 5:48 PM, Ian Cordasco <sigmaviru...@gmail.com> wrote:
> -Original Message-
> From: Thomas Herve <the...@redhat.com>
[snip]
>> Respecting the guidelines is totally fair, but review stats won't tell
>> you much, at least in my case: I barely do any stable reviews because
>> I don't have approve rights. In the case of Heat, 90% of the backports
>> are without conflicts, so stable reviews are just about verifying the
>> guidelines and that the patch matches what's in master.
>>
>> But, I've been working on Heat for 4 years, I made about 1400 reviews
>> on it, and I've been PTL. And the same for the other people that Zane
>> mentioned. I feel we should be trusted on stable branches.
>
> That seems like a very poor excuse - "I can't approve so I don't
> review". I'm a stable maintenance core because I was reviewing stable
> branch changes first. I had a good track record, and both the existing
> Glance stable maint core reviewers and the global team agreed I had
> displayed sound judgment for those.

It's not an excuse, I'm explaining why I don't do many stable reviews.
My time is valuable, and I don't do all the reviews that I could on
master already. I'd rather spend review time where I can move the
needle, instead on patches where ultimately that won't matter. If I
see a stable patch which doesn't make sense, I'll comment, but it's
very rare. On others, if that looks fine I don't do anything. Because
most contributors (on Heat at least) already made the effort to think
whether their change was backport worthy. And I don't chase stats.

> Without being able to assess the quality of your reviews, how should
> anyone else trust you with the stability of those branches?

You can assess the quality of my reviews on master. I don't see how
stable is so different. We can't break APIs, we can't change the DB
randomly, we can't break compatibility. The pain points (dependencies,
config options, features) are most of the time easy to spot.

(Also, Matt mentioned review stats. I could have 200 stable +1, that
would maybe look nice on paper, but not prove anything, if there is
anything to prove).

>> > There are reviewstats tools for seeing the stable review numbers for Heat, 
>> > I
>> > haven't run that though to check against those proposed above, but it's
>> > probably something I'd do first before just adding a bunch of people.
>>
>> I appreciate your guidance and input, but shouldn't we decide our
>> stable maintainers, the same way we decide cores? The current list
>> contains at least one person that doesn't contribute anymore, so it's
>> not like it's super curated.
>
> This is how every other service team works (Nova, Keystone, Glance,
> etc.). Just because the global stable maint team hasn't removed an
> inactive person doesn't invalidate their assessment of potential core
> reviewers.

Saying that how other work is not a fantastic argument. We'd need to
know if that actually works for them.

At any rate, it's a matter of trust, a subject that comes from time to
time, and it's fairly divisive. In this case though, I find it ironic
that I can approve whatever garbage I want on master, it can make its
way into a release, but if I want a bugfix backported into another
branch, someone else has to supervise me.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable][heat] Heat stable-maint additions

[Thomas Herve]

On Fri, Feb 17, 2017 at 4:14 PM, Matt Riedemann <mriede...@gmail.com> wrote:
> On 2/15/2017 12:40 PM, Zane Bitter wrote:
>>
>> Traditionally Heat has given current and former PTLs of the project +2
>> rights on stable branches for as long as they remain core reviewers.
>> Usually I've done that by adding them to the heat-release group.
>>
>> At some point the system changed so that the review rights for these
>> branches are no longer under the team's control (instead, the
>> stable-maint core team is in charge), and as a result at least the
>> current PTL (Rico Lin) and the previous PTL (Rabi Mishra), and possibly
>> others (Thomas Herve, Sergey Kraynev), haven't been added to the group.
>> That's slowing down getting backports merged, amongst other things.
>>
>> I'd like to request that we update the membership to be the same as
>> https://review.openstack.org/#/admin/groups/152,members
>>
>> Rabi Mishra
>> Rico Lin
>> Sergey Kraynev
>> Steve Baker
>> Steven Hardy
>> Thomas Herve
>> Zane Bitter
>>
>> I also wonder if the stable-maint team would consider allowing the Heat
>> team to manage the group membership again if we commit to the criteria
>> above (all current/former PTLs who are also core reviewers) by just
>> adding that group as a member of heat-stable-maint?
>>
>> thanks,
>> Zane.
>>
>
> Reviewing patches on stable branches have different guidelines, expressed
> here [1]. In the past when this comes up I've asked if the people being
> asked to be added to the stable team for a project have actually been doing
> reviews on the stable branches to show they are following the guidelines,
> and at times when this has come up the people proposed (usually PTLs)
> haven't, so I've declined at that time until they start actually doing
> reviews and can show they are following the guidelines.

Respecting the guidelines is totally fair, but review stats won't tell
you much, at least in my case: I barely do any stable reviews because
I don't have approve rights. In the case of Heat, 90% of the backports
are without conflicts, so stable reviews are just about verifying the
guidelines and that the patch matches what's in master.

But, I've been working on Heat for 4 years, I made about 1400 reviews
on it, and I've been PTL. And the same for the other people that Zane
mentioned. I feel we should be trusted on stable branches.

> There are reviewstats tools for seeing the stable review numbers for Heat, I
> haven't run that though to check against those proposed above, but it's
> probably something I'd do first before just adding a bunch of people.

I appreciate your guidance and input, but shouldn't we decide our
stable maintainers, the same way we decide cores? The current list
contains at least one person that doesn't contribute anymore, so it's
not like it's super curated.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][glance][glare][all] glance/glare/artifacts/images at the PTG

[Thomas Herve]

On Mon, Feb 13, 2017 at 8:39 PM, Clint Byrum  wrote:

[snip]

> Any attempt at versioning that is not git, will frustrate any git user.
> This cat's already out of the bag, but I'd suggest adding git repositories
> as a blob container type and finding a way to allow git to push/pull
> to/from swift. That would be an amazing feature _for swift_ anyway
> (maybe it already exists?) but it would allow Glare to piggy back on all
> of the collective versioning capabilities in Git rather than having to
> chase git.

That has been done:
https://blogs.rdoproject.org/6642/openstack-swift-as-backend-for-git-part-1

I don't know if it's maintained nowadays, but I suspect it could be
picked up if interest raises.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Attempting to proxy websockets through Apache or HAProxy for Zaqar

[Thomas Herve]

On Tue, Jan 17, 2017 at 6:23 PM, Dan Trainor  wrote:
> Hi -
>
> In an attempt to work on [0], I've been playing around with proxying all the
> service API endpoints that the UI needs to communicate with, through either
> haproxy or Apache to avoid a bug[1] around how non-Chrome browsers handle
> SSL connections to different ports on the same domain.
>
> The blueprint suggests using haproxy for this, but we're currently using the
> "old" notation of listen/server, not frontend/backend.  The distinction is
> important because the ACLs that would allow any kind of proxying to
> facilitate this are only available in the latter notation.  In order to do
> this in haproxy, tripleo::haproxy would need a rewrite (looks pretty
> trivial, but likely out of scope for this).  So I'd really like to isolate
> this to UI, which is convenient since UI runs largely self-contained inside
> Apache.
>
> I've made some good progress with most all of the services, since they were
> pretty straight-forward - mod_proxy handles them just fine.  The one I'm not
> able to make work right now is the websocket service that UI uses.
> Ultimately, I see the Websocket connection get upgraded and the Websocket
> opens, but stays open indefinitely and will never see more than 0 bytes.  No
> data is transferred from the browser over the Websocket.  This connection
> hangs indefinitely, and UI does not complete any operations that depend on
> the Zaqar Websocket.
>
> Observing trace6[4] output, I can see mod_proxy_wstunnel (which relies on
> mod_proxy) make the connection, I can see Zaqar recognize the request in
> logs, the client (UI) doesn't send or receive any data from it.  It's as if
> immediately after the Upgrade[2], the persistent Websocket connection just
> dies.
>
> I've had limited success using a couple different implementations of this in
> Apache.  ProxyPass/ProxyPassReverse looks as if it should work (so long as
> mod_proxy_wstunnel is loaded), but this is not my experience.  Using a
> mod_rewrite rule[3] to force the specific Websocket proxy for a specific URI
> (/zaqar) has the same outcome.
>
> In its most simple form, the ProxyPass rule I'm attempting to use is:
>
>   ProxyPass "/zaqar""ws://192.0.2.1:9000/"
>   ProxyPassReverse  "/zaqar""ws://192.0.2.1:9000/"

I tried that configuration, and it worked fine with a python websocket
client. I created a queue, a subscription, and was able to retrieve
notifications properly. I used Apache 2.4.23.

Is it possible that you're having a CORS issue?

> Here's Zaqar's Websocket transport answering the request, creating both a
> queue and a subscription but no data after that:

What do you mean by "no data after that"? What kind of data are you
expecting after? Just getting those 2 messages seems to indicate that
it works fine to me. Are you getting timeouts, unexpected closed
connections?

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] 答复: [heat] glance v2 support?

[Thomas Herve]

On Wed, Jan 11, 2017 at 3:34 PM, Emilien Macchi <emil...@redhat.com> wrote:
> On Wed, Jan 11, 2017 at 2:50 AM, Thomas Herve <the...@redhat.com> wrote:
>> I think this is going where I thought it would: let's not do anything.
>> The image resource is there for v1 compatibility, but there is no
>> point to have a v2 resource, at least right now.
>
> If we do nothing, we force our heat-template users to keep Glance v1
> API enabled in their cloud (+ running Glance Registry service), which
> at some point doens't make sense, since Glance team asked to moved
> forward with Glance v2 API.
>
> I would really recommend to move forward and stop ignoring the new API 
> version.

Emilien was right: by defaulting to Glance v1, we still required it
for the image constraint, which is used everywhere like servers and
volumes. We can easily switch to v2 for this, I'll do it right away.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] 答复: [heat] glance v2 support?

[Thomas Herve]

On Tue, Jan 10, 2017 at 10:41 PM, Clint Byrum  wrote:
> Excerpts from Zane Bitter's message of 2017-01-10 15:28:04 -0500:
>> location is a required property:
>>
>> http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Glance::Image
>>
>> The resource type literally does not do anything else but expose a Heat
>> interface to a feature of Glance that no longer exists in v2. That's
>> fundamentally why "add v2 support" has been stalled for so long ;)
>>
>
> I think most of this has been beating around the bush, and the statement
> above is the heart of the issue.
>
> The functionality was restricted and mostly removed from Glance for a
> reason. Heat users will have to face that reality just like users of
> other orchestration systems have to.
>
> If a cloud has v1.. great.. take a location.. use it. If they have v2..
> location explodes. If you want to get content in to that image, well,
> other systems have to deal with this too. Ansible's os_image will upload
> a local file to glance for instance. Terraform doesn't even include
> image support.
>
> So the way to go is likely to just make location optional, and start
> to use v2 when the catalog says to. From there, Heat can probably help
> make the v2 API better, and perhaps add support to to the Heat API to
> tell the user where they can upload blobs of data for Heat to then feed
> into Glance.

Making location optional doesn't really make sense. We don't have any
mechanism in a template to upload data, so it would just create an
empty shell that you can't use to boot instances from.

I think this is going where I thought it would: let's not do anything.
The image resource is there for v1 compatibility, but there is no
point to have a v2 resource, at least right now.

We could document how to hide the resource in Heat if you don't deploy
Glance v1.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] 答复: [heat] glance v2 support?

[Thomas Herve]

On Tue, Jan 10, 2017 at 9:28 PM, Zane Bitter  wrote:
> On 10/01/17 14:17, Tim Bell wrote:
>>
>>
>>> On 10 Jan 2017, at 17:41, Zane Bitter >> > wrote:
>>>
>>> On 10/01/17 05:25, Flavio Percoco wrote:



> I'd recommend Heat to not use locations as that will require deployers
> to either enable them for everyone or have a dedicate glance-api node
> for Heat.
> If not use location, do we have other options for user? What
> should user to do before create a glance image using v2? Download the
> image data? And then pass the image data to glance api? I really don't
> think it's good way.
>

 That *IS* how users create images. There used to be copy-from too (which
 may or
 may not come back).

 Heat's use case is different and I understand that but as I said in my
 other
 email, I do not think sticking to v1 is the right approach. I'd rather
 move on
 with a deprecation path or compatibility layer.
>>>
>>>
>>> "Backwards-compatibility" is a wide-ranging topic, so let's break this
>>> down into 3 more specific questions:
>>>
>>> 1) What is an interface that we could support with the v2 API?
>>>
>>> - If copy-from is not a thing then it sounds to me like the answer is
>>> "none"? We are not ever going to support uploading a multi-GB image
>>> file through Heat and from there to Glance.
>>> - We could have an Image resource that creates a Glance image from a
>>> volume. It's debatable how useful this would be in an orchestration
>>> setting (i.e. in most cases this would have to be part of a larger
>>> workflow anyway), but there are some conceivable uses I guess. Given
>>> that this is completely disjoint from what the current resource type
>>> does, we'd make it easier on everyone if we just gave it a new name.
>>>
>>> 2) How can we avoid breaking existing stacks that use Image resources?
>>>
>>> - If we're not replacing it with anything, then we can just mark the
>>> resource type as first Deprecated, and then Hidden and switch the back
>>> end to use the v2 API for things like deleting. As long as nobody
>>> attempts to replace the image then the rest of the stack should
>>> continue to work fine.
>>>
>>
>> Can we only deprecate the resources using the location function but
>> maintain backwards compatibility if the location function is not used?
>
>
> location is a required property:
>
> http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Glance::Image
>
> The resource type literally does not do anything else but expose a Heat
> interface to a feature of Glance that no longer exists in v2. That's
> fundamentally why "add v2 support" has been stalled for so long ;)

Throwing stuff against the wall, but could we solve the issue in
heatclient? If we change it to handle the location property, upload
the image from the client, and pass the id to Heat, it could be
somewhat transparent to the user. We'd need to do it in Horizon
though. For the heatclient as a library it's not perfect, but it may
be good enough.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][tripleo] Heat memory usage in the TripleO gate during Ocata

[Thomas Herve]

On Fri, Jan 6, 2017 at 6:12 PM, Zane Bitter  wrote:
> tl;dr everything looks great, and memory usage has dropped by about 64%
> since the initial Newton release of Heat.
>
> I re-ran my analysis of Heat memory usage in the tripleo-heat-templates
> gate. (This is based on the gate-tripleo-ci-centos-7-ovb-nonha job.) Here's
> a pretty picture:
>
> https://fedorapeople.org/~zaneb/tripleo-memory/20170105/heat_memused.png
>
> There is one major caveat here: for the period marked in grey where it says
> "Only 2 engine workers", the job was configured to use only 2 heat-enginer
> worker processes instead of 4, so this is not an apples-to-apples
> comparison. The inital drop at the beginning and the subsequent bounce at
> the end are artifacts of this change. Note that the stable/newton branch is
> _still_ using only 2 engine workers.
>
> The rapidly increasing usage on the left is due to increases in the
> complexity of the templates during the Newton cycle. It's clear that if
> there has been any similar complexity growth during Ocata, it has had a tiny
> effect on memory consumption in comparison.

Thanks a lot for the analysis. It's great that things haven't gotten off track.

> I tracked down most of the step changes to identifiable patches:
>
> 2016-10-07: 2.44GiB -> 1.64GiB
>  - https://review.openstack.org/382068/ merged, making ResourceInfo classes
> more memory-efficient. Judging by the stable branch (where this and the
> following patch were merged at different times), this was responsible for
> dropping the memory usage from 2.44GiB -> 1.83GiB. (Which seems like a
> disproportionately large change?)

Without wanting to get the credit, I believe
https://review.openstack.org/377061/ is more likely the reason here.

>  - https://review.openstack.org/#/c/382377/ merged, so we no longer create
> multiple yaql contexts. (This was responsible for the drop from 1.83GiB ->
> 1.64GiB.)
>
> 2016-10-17: 1.62GiB -> 0.93GiB
>  - https://review.openstack.org/#/c/386696/ merged, reducing the number of
> engine workers on the undercloud to 2.
>
> 2016-10-19: 0.93GiB -> 0.73GiB (variance also seemed to drop after this)
>  - https://review.openstack.org/#/c/386247/ merged (on 2016-10-16), avoiding
> loading all nested stacks in a single process simultaneously much of the
> time.
>  - https://review.openstack.org/#/c/383839/ merged (on 2016-10-16),
> switching output calculations to RPC to avoid almost all simultaneous
> loading of all nested stacks.
>
> 2016-11-08: 0.76GiB -> 0.70GiB
>  - This one is a bit of a mystery???

Possibly https://review.openstack.org/390064/ ? Reducing the
environment size could have an effect.

> 2016-11-22: 0.69GiB -> 0.50GiB
>  - https://review.openstack.org/#/c/398476/ merged, improving the efficiency
> of resource listing?
>
> 2016-12-01: 0.49GiB -> 0.88GiB
>  - https://review.openstack.org/#/c/399619/ merged, returning the number of
> engine workers on the undercloud to 4.
>
> It's not an exact science because IIUC there's a delay between a patch
> merging in Heat and it being used in subsequent t-h-t gate jobs. e.g. the
> change to getting outputs over RPC landed the day before the
> instack-undercloud patch that cut the number of engine workers, but the
> effects don't show up until 2 days after. I'd love to figure out what
> happened on the 8th of November, but I can't correlate it to anything
> obvious. The attribution of the change on the 22nd also seems dubious, but
> the timing adds up (including on stable/newton).
>
> It's fair to say that none of the other patches we merged in an attempt to
> reduce memory usage had any discernible effect :D
>
> It's worth reiterating that TripleO still disables convergence in the
> undercloud, so these are all tests of the legacy code path. It would be
> great if we could set up a non-voting job on t-h-t with convergence enabled
> and start tracking memory use over time there too. As a first step, maybe we
> could at least add an experimental job on Heat to give us a baseline?

+1. We haven't made any huge changes into that direction, but having
some info would be great.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates

[Thomas Herve]

On Fri, Dec 16, 2016 at 2:57 PM, Steven Hardy <sha...@redhat.com> wrote:
> On Fri, Dec 16, 2016 at 02:03:10PM +0100, Thomas Herve wrote:
>> On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente <gfide...@redhat.com> wrote:
>> > I was wondering if it would make sense to have a property for the existing
>> > Workflow resource to let the user decide if the workflow should *also* be
>> > triggered on CREATE/UPDATE? And if it would make sense to block the 
>> > Workflow
>> > resource until the execution result is returned in that case?
>>
>> I'm not super in favor of that. It's conflicting 2 different concepts here.
>
> Well, they're already conflated in the mistral workflow resource, because
> we allow creating an execution by sending a signal:
>
> https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/mistral/workflow.py#L586

Right, I mentioned that elsewhere. But it doesn't change the resource
interface, for better or worse.

> That satisfies the requirement for asynchronous (signal/alarm driven)
> workflow execution, but we need the synchronous version that can be fully
> integrated into the heat stack lifecycle (without external dependencies on
> alarms etc).
>
> I know we've previously tried to steer execute/run type actions to signal
> driven interfaces (and I myself have opposed these kinds of resources in
> the past, to be honest).  However, I can't currently see a better way to
> handle this requirement, and given it may be pretty easy to fix (refactor
> handle_signal and add a boolean to each handle_foo function), I think this
> discussion warrants revisiting.

It's unclear what changed since the discussion happened, just that you
have a use case without another approach possible?

>> > Alternatively, would an ex-novo Execution resource make more sense?
>>
>> We had some discussions here: https://review.openstack.org/267770.
>> Executing things as part of a template is a tricky proposition. I
>> think we'd like it to be more akin to software deployments, where it
>> runs on actions. We also were talking about doing something like AWS
>> CustomResource in Heat, which may look like WorkflowExecution (at
>> least one part of it).
>
> Yeah I think whichever approach we take, a list of actions similar to
> SoftwareDeployment makes sense, then you can elect to run a specific
> workflow at any state transition in the lifecycle.

Among the various solutions, I think that's the one I like the best
for now. It doesn't touch the workflow resource interface, and it
seems to fit relatively naturally (an API to call, a state to check,
etc).

>> > Or are there different ideas, approaches to the problem?
>>
>> If you could define the event outside of Heat (in your example,
>> publish something when a swift node is available), then you could use
>> Zaqar to trigger your workflow. If you want Heat to block that won't
>> do it though.
>
> Yeah that doesn't solve our use-case, we want to run a workflow during an
> overcloud stack create, wait for the result, then continue (or fail).

Noted.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][TripleO] How to run mistral workflows via templates

[Thomas Herve]

On Fri, Dec 16, 2016 at 1:17 PM, Giulio Fidente  wrote:
> hi,
>
> we're trying to address in TripleO a couple of use cases for which we'd like
> to trigger a Mistral workflow from a Heat template.
>
> One example where this would be useful is the creation of the Swift rings,
> which need some data related to the Heat stack (like the list of Swift nodes
> and their disks), so it can't be executed in advance, yet it provides data
> which is needed to complete successfully the deployment of the overcloud.
>
> Currently we can create a workflow from Heat, but we can't trigger its
> execution and also we can't block Heat on the result of the execution.

You can trigger it out of band with resource signal. But you're right,
Heat won't wait for the result.

> I was wondering if it would make sense to have a property for the existing
> Workflow resource to let the user decide if the workflow should *also* be
> triggered on CREATE/UPDATE? And if it would make sense to block the Workflow
> resource until the execution result is returned in that case?

I'm not super in favor of that. It's conflicting 2 different concepts here.

> Alternatively, would an ex-novo Execution resource make more sense?

We had some discussions here: https://review.openstack.org/267770.
Executing things as part of a template is a tricky proposition. I
think we'd like it to be more akin to software deployments, where it
runs on actions. We also were talking about doing something like AWS
CustomResource in Heat, which may look like WorkflowExecution (at
least one part of it).

> Or are there different ideas, approaches to the problem?

If you could define the event outside of Heat (in your example,
publish something when a swift node is available), then you could use
Zaqar to trigger your workflow. If you want Heat to block that won't
do it though.


-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][sahara][magnum][tripleo] Scaling nested stack validation

[Thomas Herve]

On Wed, Nov 23, 2016 at 11:58 PM, Zane Bitter  wrote:
> We discussed $SUBJECT at the summit as one of the main performance problems
> that people are running into when trying to create very large autoscaling
> groups, as projects like Sahara, Magnum, TripleO, OpenShift are wont to do.
> Of course, as we all know, validation happens synchronously, so it's prone
> to causing RPC timeouts that mean a hard failure of the parent stack.
>
> First the good news - I just committed this patch:
>
> https://review.openstack.org/#/c/400961/
>
> which should mean from now on that resources with identical definitions will
> not all be validated, and instead we'll just validate one representative
> one. In theory this should mean that autoscaling groups should now validate
> in constant rather than linear time. If anyone from one of the affected
> projects is able to confirm this, then I'd be happy to backport the patch to
> stable/newton. It really is very simple.
>
> The bad news here is for users of ResourceGroups with %index% substitution
> (*cough*TripleO*cough*) - this makes each resource definition unique, so it
> won't benefit from this fix. (Adding this to my mental list of reasons why
> index substitution is bad.)
>
>
> I also investigated another issue, which is that since the fix for
> https://bugs.launchpad.net/heat/+bug/1388140 landed (in Kilo) I believe we
> are validating nested stacks multiple times (specifically, m times, where m
> is the stack's depth in the tree):
>
>   root childgrandchild
>
>   create
>-> validate --> validate --> validate
>-> Resource.create ===> create
> -> validate --> validate
> -> Resource.create ===> create
>  -> validate
>
> The only good news here is that ResourceGroup is smart enough to make sure
> that it generates a nested stack with at most 1 resource to validate when
> validate() is called. (However, when the nested stack is created, and thus
> validated, it is of course full-sized.) Autoscaling groups make no such
> allowances, but the patch above should actually have the same effect. (We
> can't get rid of the special case for ResourceGroup though, because of index
> substitution.)
>
> An obvious fix would be to disable validation - or, more specifically,
> validation of _resources_ - on create/update for stacks that have a non-null
> owner_id (i.e. nested stacks), so that we had something like:
>
>   root childgrandchild
>
>   create
>-> validate --> validate --> validate
>-> Resource.create ===> create
> -> Resource.create ===> create
>
> That would eliminate the duplication/triplication/multiplication of
> validation. It would also mean that we'd cut out the expensive part of
> ResourceGroup validation with index substitution, leaving only the cheap
> part.
>
> One downside is that in the ResourceGroup/index substitution case we'd be
> creating resources whose definitions hadn't _ever_ been validated. I _think_
> that's safe, in the sense that you'd just hear about errors later, as
> opposed to everything falling over in a heap, but it's difficult to be
> certain. Hearing about problems late is also not ideal (since it may cause
> otherwise-healthy siblings to be cancelled), but I would guess that heavy
> users like TripleO developers would say that it's worth the tradeoff.
>
> However, one other thing about this bothers me. The part of validation that
> we're keeping:
>
>-> validate --> validate --> validate
>
> involves loading all of the nested stacks in memory at once (i.e. the thing
> we were not supposed to be doing any more in Kilo, in favour of farming
> nested stacks out over RPC.) As we discovered when we found out we were
> doing the same thing with outputs[1], this is a bit like hanging out a giant
> "Kick Me" sign for the OOM Killer.
>
> That's mitigated quite a lot by my patch though... we'll load the whole
> autoscaling group stack in memory, but if its members are themselves nested
> stacks we'll load only one of them. So the scaling tendencies will hopefully
> be dominated by the complexity of your templates more than than the size of
> your deployment. ResourceGroup is in a better position, because its nested
> stack will actually have only one member, so the size shouldn't affect
> memory consumption at all during validation.
>
> Some options:
> 1) Chalk it up to an acceptable tradeoff
> 2) Add a single-member special case for autoscaling group validation
> 3) Farm out the nested validation over RPC
> 4) Both (2) & (3)
> 5) Some totally different arrangement of how nested stacks are validated

I think I'd like to see what difference 3 makes. Maybe then also do 2.
Again, we really need to have some reproducible big template that 

[openstack-dev] [heat][tripleo] New repository for software-config elements

[Thomas Herve]

Hi all,

Historically elements to create images using software config were
developed in the heat-templates repository, which turned out to mean
that this had to be packaged, etc. Today we were asked if tags could
be added to help maintaining the packages. Before we do that, I wonder
if we should extract the elements in a different repository. We
already have tests which are only applicable to this specific subset
of the repo, so it shouldn't be too hard.

In summary: let's create a new repository
heat-software-config-elements, and move everything from
hot/software-config/elements/ in the heat-templates repository to it
(and the associated tests).

Thoughts?

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][zaqar][telemetry] Subscribing to events

[Thomas Herve]

On Tue, Oct 25, 2016 at 12:12 AM, Zane Bitter <zbit...@redhat.com> wrote:
> On 22/10/16 10:38, Thomas Herve wrote:
>>
>> Hi all,
>>
>> One of my long time goal since I started contributing to OpenStack is
>> to try to remove polling where I can. With Zaqar WebSocket support, we
>> now have a transport available for users to connect to, and where we
>> can push notifications. We already leveraged that in Heat [1], so that
>> you can manipulate a stack and you'll be notified when its status
>> changes.
>>
>> Still, not everyone uses Heat, and under the hood it still polls for
>> you. We should be able to use the various notifications that projects
>> publish to push similar events. Ceilometer already consumes those
>> notifications and try to make them somewhat consumable [2].
>>
>> The missing piece is something that bridges Ceilometer and Zaqar. I
>> wrote a small service [3] which provide a simple API, so that you can
>> say "Subscribe to those events and send them to this queue". The data
>> flow looks like this:
>>
>> Service (Nova, Neutron) -> Notification -> Ceilometer -> Bridge ->
>> Zaqar -> User.
>>
>> This way, you'll get a Zaqar message per event, with some filtering
>> done by the bridge service. It's far from being ideal, as the
>> notification format is a endless topic of conversation, but I hope
>> that if we start using it things will move further. I also hope I can
>> get some feedback on the approach.
>
>
> Could you document what makes this different from
> http://docs.openstack.org/developer/aodh/event-alarm.html? (I can think of
> some ways, but it's not clear to me whether a separate service is the right
> thing or if the existing Aodh event alarms can be modified to do what we
> want.)

Asking the tough questions :). I don't know about the details, but
it's possible that since https://review.openstack.org/#/c/335289/ got
merged, you have the same possibilities nowadays. I'd need to test it,
in which case my service is not necessary anymore (though the other
questions still apply). But in terms of semantics, it feels a bit
weird to use alarming for continuous event retrieval.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [heat][zaqar][telemetry] Subscribing to events

[Thomas Herve]

Hi all,

One of my long time goal since I started contributing to OpenStack is
to try to remove polling where I can. With Zaqar WebSocket support, we
now have a transport available for users to connect to, and where we
can push notifications. We already leveraged that in Heat [1], so that
you can manipulate a stack and you'll be notified when its status
changes.

Still, not everyone uses Heat, and under the hood it still polls for
you. We should be able to use the various notifications that projects
publish to push similar events. Ceilometer already consumes those
notifications and try to make them somewhat consumable [2].

The missing piece is something that bridges Ceilometer and Zaqar. I
wrote a small service [3] which provide a simple API, so that you can
say "Subscribe to those events and send them to this queue". The data
flow looks like this:

Service (Nova, Neutron) -> Notification -> Ceilometer -> Bridge ->
Zaqar -> User.

This way, you'll get a Zaqar message per event, with some filtering
done by the bridge service. It's far from being ideal, as the
notification format is a endless topic of conversation, but I hope
that if we start using it things will move further. I also hope I can
get some feedback on the approach.

One question is whether we want to skip the subscription phase, and
simply makes all those events available to users at a known place (for
example, all events in the Zaqar queue events-queue, or maybe all Nova
events in the compute-events-queue). That pushes filtering to the
user, but it also simplifies the bootstrap phase.

Anyway, I wanted to send this before the summit so that people
interested can come talk to me.

Thanks,

-- 
Thomas

[1] 
http://docs.openstack.org/developer/heat/template_guide/environment.html#retrieving-events

[2] 
http://docs.openstack.org/developer/ceilometer/events.html#events-from-notifications

[3] https://github.com/therve/nabu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat]Delete trust with exception when deleting a stack?

[Thomas Herve]

On Fri, Oct 14, 2016 at 4:39 PM, zhu4236926  wrote:
> Hi guys,
>1. I set the conifg deferred_auth_method with value trusts, so a trust
> record will be created in keystone when creating a stack, and heat save the
> trust result  to db.
>2. Delete the stack, if all resources has been deleted successfully, heat
> will delete the trust record in keytstone, then clean the db record in heat,
> if an exception occrured when deleting the trust in keystone, heat just
> print a error log and ignore the exception, then set the state of stack to
> DELETE FAILED, but the db record in heat has been deleted,  though the trust
> record is still in keystone.
>3. Delete the stack again, because all resource and trust record in heat
> has beened deleted, the stack delete successfully this time,  the trust in
> keystone created by the stack is still exsit.
>
>   My heat version is Mitaka, I think this may be a BUG,  if it's just a
> correct desigin in heat, could any one help me?

Hi,

We've had issues around that area, so it's possible there is a bug,
but what you describe looks mostly like correct behavior. Unless the
keystone error was transient, and we're meant to retry. What kind of
errors do you get exactly?

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] resigning from heat-cores

[Thomas Herve]

On Mon, Sep 12, 2016 at 2:35 PM, Pavlo Shchelokovskyy
 wrote:
> Hi Heaters,
>
> with great regret I announce my resignation from the heat-core team.
>
> About a year ago I was reassigned to another project, and despite my best
> efforts I came to conclusion that unfortunately I can not keep up with
> duties expected from Heat core team member in appropriate capacity.
>
> I do still work on OpenStack, so I'm not leaving the community altogether,
> and will be available in e.g. IRC. I also have some ideas left to implement
> in Heat, but, given the great community we've built around the project, I
> could surely make it as an ordinary contributor.
>
> It was an honor to be a member of this team, I’ve learned a lot during this
> time. Hope to see some of you in Barcelona :)

Thanks a ton Pavlo, it's great working with you, and we'll hope to see
you around again.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] Convergence status

[Thomas Herve]

Hi all,

We're approaching release time, and there are still some concerns
whether or not we can make convergence the default. In particular we
have a somewhat longstanding issue with canceling creation [1]. The
good news is that this is almost fixed [2], and update cancel is
almost there too [3]. We expect to see them merged by RC1 next week,
so please review those if you can.

I also have some confidence overall as projects like Magnum and Sahara
have been running tests with convergence enabled for about 3 months
now, and no particular issues have been reported about it.

That said, if we can't manage to get those in by RC1, we'll have to
revert back to legacy. Those behaviors are too important to release
without them. Hopefully, we'll make it work, and we can focus on
making TripleO using convergence during the next cycle.

Thanks,

-- 
Thomas


[1] https://bugs.launchpad.net/heat/+bug/1592374
[2] https://review.openstack.org/#/c/354000/
[3] https://review.openstack.org/#/c/362346/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] what permission is required to create a Keystone trust

[Thomas Herve]

On Thu, Sep 1, 2016 at 5:54 AM, Matt Jia  wrote:
> Hi,
>
> I am experimenting the Keystone Trusts feature with a script which creates a
> trust between two users.
>
> import keystoneclient.v3 as keystoneclient
> #import swiftclient.client as swiftclient
>
>
> auth_url_v3 = 'http:/xxxt.com:5000/v3/'
>
>
> demo = keystoneclient.Client(auth_url=auth_url_v3,
>  username='demo',
>  password='openstack',
>  project='demo')
> import pdb; pdb.set_trace()
> alt_demo = keystoneclient.Client(auth_url=auth_url_v3,
>  username='alt_demo',
>  password='openstack',
>  project='alt_demo')
>
> trust = demo.trusts.create(trustor_user=demo.user_id,
>trustee_user=alt_demo.user_id,
>project=demo.tenant_id)

I believe you need to at least specify one role to be able to create a
trust. You can't delegate all users roles if you pass in a project.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][yaql] Deep merge map of lists?

[Thomas Herve]

On Tue, Aug 30, 2016 at 6:02 PM, Steven Hardy  wrote:
> On Tue, Aug 30, 2016 at 04:10:47PM +0200, Jiří Stránský wrote:
>>
>> On 30.8.2016 10:17, Steven Hardy wrote:
>>
>> 
>>
>> > Yeah, that gets us closer, but we do need to handle more than one value
>> > (list entry) per key, e.g:
>> >
>> >  data:
>> >l:
>> >  - "gnocchi_metricd_node_names": ["a0", "a1", "a2"]
>> >"tripleo_packages_node_names": ["a0", "a1", "a2"]
>> >  - "nova_compute_node_names": ["b0"]
>> >"tripleo_packages_node_names": ["b0"]
>> >
>> > Output needs to be like:
>> >
>> >  "gnocchi_metricd_node_names": ["a0", "a1", "a2"]
>> >  "tripleo_packages_node_names": ["a0", "a1", "a2", "b0"]
>> >  "nova_compute_node_names": ["b0"]
>> >
>>
>> Hoping this could do it:
>>
>> [stack@instack ~]$ cat yaq.yaml
>> heat_template_version: 2016-10-14
>>
>> outputs:
>>   debug:
>> value:
>>   yaql:
>> expression: $.data.l.reduce($1.mergeWith($2))
>> data:
>>   l:
>> - "gnocchi_metricd_node_names": ["a0", "a1", "a2"]
>>   "tripleo_packages_node_names": ["a0", "a1", "a2"]
>> - "nova_compute_node_names": ["b0"]
>>   "tripleo_packages_node_names": ["b0"]
>
> Thanks for this!
>
> Unfortunately I dont think it works with more than two list items:
>
>   debug_tripleo2:
> value:
>   yaql:
> expression: $.data.l.reduce($1.mergeWith($2))
> data:
>   l:
> - "gnocchi_metricd_node_names": ["overcloud-controller-0",
>   "overcloud-controller-1", "overcloud-controller-2"]
>   "tripleo_packages_node_names": ["overcloud-controller-0", 
> "overcloud-controller-1", "overcloud-controller-2"]
> - "nova_compute_node_names": ["overcloud-compute-0"]
>   "tripleo_packages_node_names": ["overcloud-compute-0"]
>   "tripleo_packages_node_names2": ["overcloud-compute-0"]
> - "ceph_osd_node_names": ["overcloud-cephstorage-0"]
>   "tripleo_packages_node_names": ["overcloud-cephstorage-0"]
>   "tripleo_packages_node_names2": ["overcloud-cephstorage-0"]
>
> $ heat output-show foo5 debug_tripleo2
> stack output show" instead
> Output error: can only concatenate tuple (not "list") to tuple
>
> I've not dug too deeply yet, but assuming that's a yaql error vs a heat bug
> it looks like it won't work.

I'd say it's a yaql bug (commented on the bug opened in Heat), that
can work around using the list_merger argument:

$.data.l.reduce($1.mergeWith($2, listMerger => $1.toList() + $2.toList()))

Still slightly more elegant than the one I came up with.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][yaql] Deep merge map of lists?

[Thomas Herve]

On Mon, Aug 29, 2016 at 3:16 PM, Steven Hardy <sha...@redhat.com> wrote:
> On Mon, Aug 29, 2016 at 07:07:09AM +0200, Thomas Herve wrote:
>> dict($.groupBy($.keys().toList()[0], $.values().toList()[0][0]))
>>
>> ought to work, I believe?
>
> So, as it turns out, my example above was bad, and groupBy only works if
> you have a list of maps with exactly one key, we actually need this:
>
>   # Example of tripleo format
>   # We need an output of
>   # "gnocchi_metricd_node_names": ["overcloud-controller-0"]
>   # "tripleo_packages_node_names": ["overcloud-controller-0", 
> "overcloud-compute-0"]
>   # "nova_compute_node_names": ["overcloud-compute-0"]
>   debug_tripleo:
> value:
>   yaql:
> expression: dict($.data.l.groupBy($.keys().toList()[0], 
> $.values().toList()[0][0]))
> data:
>   l:
> - "gnocchi_metricd_node_names": ["overcloud-controller-0"]
>   "tripleo_packages_node_names": ["overcloud-controller-0"]
> - "nova_compute_node_names": ["overcloud-compute-0"]
>   "tripleo_packages_node_names": ["overcloud-compute-0"]
>
> So, I'm back to wondering how we make the intermediate assignement of 
> tripleo_packages_node_names

Well I didn't know all the constraints :).

$.selectMany($.items()).groupBy($[0], $[1][0])

is another attempt. It won't work if you have more than one value per
key in the original data, but I think it will handle multiple keys.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat][yaql] Deep merge map of lists?

[Thomas Herve]

On Sun, Aug 28, 2016 at 11:58 PM, Steven Hardy  wrote:
> Hi all,
>
> I have a need to merge a list of maps of lists:
>
> heat_template_version: 2016-10-14
>
> outputs:
>   debug:
> value:
>   yaql:
> # dict(vms=>dict($.vms.select([$.name, $])))
> expression: dict($.data.l.select([$.keys().toList()[0],
> $.values().toList()[0]]))
> data:
>   l:
> - a: [123]
> - b: [123]
> - a: [456]
>
>
>
> I want to end up with debug as:
>
>   a: [123, 456]
>   b: [123]
>
> Perhaps we need a map_deep_merge function, but can this be done with yaql?
>
> I suspect it can, but can't currently figure out how the assignment to the
> intermediate "a" value is supposed to work, any ideas on the cleanest
> approach appreciated!

I believe you don't need the intermediate value, and can rely on what
you'd do in Python with setdefault:

dict($.groupBy($.keys().toList()[0], $.values().toList()[0][0]))

ought to work, I believe?

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] a new Undercloud install driven by Heat

[Thomas Herve]

On Thu, Aug 4, 2016 at 11:48 PM, Dan Prince  wrote:
> Last week I started some prototype work on what could be a new way to
> install the Undercloud. The driving force behind this was some of the
> recent "composable services" work we've done in TripleO so initially I
> called in composable undercloud. There is an etherpad here with links
> to some of the patches already posted upstream (many of which stand as
> general imporovements on their own outside the scope of what I'm
> talking about here).
>
> https://etherpad.openstack.org/p/tripleo-composable-undercloud
>
> The idea in short is that we could spin up a small single process all-
> in-one heat-all (engine and API) and thereby avoid things like Rabbit,
> and MySQL.

I saw those patches coming, I'm interested in the all-in-one approach,
if only for testing purpose. I hope to be able to propose a solution
with broker-less RPC instead of fake RPC at some point, but it's a
good first step.

I'm a bit more intrigued by the no-auth patch. It seems that Heat
would rely heavily on Keystone interactions even after initial
authentication, so I wonder how that work. As it seems you would need
to push the same approach to Ironic, have you considered starting
Keystone instead? It's a simple WSGI service, and can work with SQLite
as well I believe.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

[Thomas Herve]

On Mon, Jul 18, 2016 at 7:13 PM, Michael Still <mi...@stillhq.com> wrote:
> On 16 Jul 2016 1:27 PM, "Thomas Herve" <the...@redhat.com> wrote:
>>
>> On Fri, Jul 15, 2016 at 8:36 PM, Fox, Kevin M <kevin@pnnl.gov> wrote:
>> > The lack of Instance Users has caused lots of projects to try and work
>> > around the lack thereof. I know for sure, Mangum, Heat, and Trove work
>> > around the lack. I'm positive others have too. As an operator, it makes me
>> > have to very carefully consider all the tradeoffs each project made, and
>> > decide if I can accept the same risk they assumed. Since each is different,
>> > thats much harder.
>>
>> Instance users would be nice. Nobody just provided a good solution. I
>> know you tried, but I don't think you succeeded. We need a good
>> implementation, easy to understand, and maybe this will move forward.
>
> I think I need a more complete definition of instance users to know what
> you're talking about here. Is this the instance locking stuff Bruno proposed
> a while ago?

I believe this is what's Kevin talks about:
https://review.openstack.org/#/c/93/

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

[Thomas Herve]

On Fri, Jul 15, 2016 at 8:36 PM, Fox, Kevin M  wrote:
> Some specific things:
>
> Magnum trying to not use Barbican as it adds an addition dependency. See the 
> discussion on the devel mailing list for details.
>
> Horizon discussions at the summit around wanting to use Zaqar for dynamic ui 
> updates instead of polling, but couldn't depend on a non widely deployed 
> subsystem.
>
> Each Advanced OpenStack Service implementing a guest controller communication 
> channel that are incompatible with each other and work around communications 
> issues differently. This makes a lot more pain for Ops to debug or architect 
> a viable solution. For example:
>  * Sahara uses ssh from the controllers to the vms. This does not play well 
> with tenant networks. They have tried to work around this several ways:
> * require every vm to have a floating ip. (Unnecessary attack surface)
> * require the controller to be on the one and only network node (Uses ip 
> netns exec to tunnel. Doesn't work for large clouds)
> * Double tunnel ssh via the controller vm's. so some vms have fips, some 
> don't. Better then all, but still not good.
>   * Trove uses Rabbit for the guest agent to talk back to the controllers. 
> This has traffic going the right direction to work well with tenant networks.
> * But Rabbit is not multitenant so a security risk if any user can get 
> into any one of the database vm's.
> Really, I believe the right solution is to use a multitenant aware message 
> queue so that the guest agent can pull in the right direction for tenant 
> networks, and not have the security risk. We have such a system already, 
> Zaqar, but its not widely deployed and projects don't want to depend on other 
> projects that aren't widely deployed.

While I agree using Barbican/Zaqar for those would be fantastic, this
is easily solved: just optionally depend on it. Heat provides features
that use Zaqar, which are not just present when Zaqar is not there.
For example, if people in the Trove project were convinced that Zaqar
was the best solution to their problem, I think they would find a way
to provide an implementation using it. I don't think they are, though.
Whatever the reasons may be.

> The lack of Instance Users has caused lots of projects to try and work around 
> the lack thereof. I know for sure, Mangum, Heat, and Trove work around the 
> lack. I'm positive others have too. As an operator, it makes me have to very 
> carefully consider all the tradeoffs each project made, and decide if I can 
> accept the same risk they assumed. Since each is different, thats much harder.

Instance users would be nice. Nobody just provided a good solution. I
know you tried, but I don't think you succeeded. We need a good
implementation, easy to understand, and maybe this will move forward.

> I'm sure there are more examples. but I hope you get I'm not just trying to 
> troll.
>
> The TC did apply inconsistant rules on letting projects in. That was for sure 
> a negative before the big tent. But it also provided a way to apply pressure 
> to projects to fix some of the issues that multiple projects face, and that 
> plague user/operators and raise the whole community up, and that has fallen 
> to the wayside since. Which is a big negative now. Maybe that could be bolted 
> on top of the Big Tent I don't know.

So I think that's the root of the issue here. You assume we can
"pressure" people do something you think is right. But that's not how
opensource works. You either implement the solution to your problem,
or convince someone else, but you don't force anybody. I think that's
what the TC recognized and changed, to the correct path. Could project
be better integrated, so that we have a more coherent "product"? Sure.
But you don't achieve that with more governance and processes.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][Senlin] Deprecation roadmap for Heat's Autoscaling resources?

[Thomas Herve]

On Mon, Jul 4, 2016 at 2:06 PM, Johannes Grassler  wrote:
> Hello,
>
> I just noticed the note at the top of
> :
>
>  | The content on this page is obsolete. The autoscaling solution is
> offloaded from Heat to Senlin since Mitaka.
>
> Right now OS::Heat::AutoScalingGroup and friends still exist, even in the
> master branch. Are they slated for removal at some stage or will they remain
> available even as Senlin becomes the go-to mechanism for autoscaling?

Hi,

They will remain available for the foreseeable future. We just don't
have any great plans to improve them currently.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Zaqar messages standardization

[Thomas Herve]

On Thu, May 26, 2016 at 11:48 AM, Jiri Tomasek <jtoma...@redhat.com> wrote:
> On 05/25/2016 08:08 PM, Thomas Herve wrote:

>> Sorry for not responding earlier, but I have some inputs. In Heat we
>> publish events on Zaqar queue, and we defined this format:
>>
>>  {
>>  'timestamp': $timestamp,
>>  'version': '0.1',
>>  'type': 'os.heat.event',
>>  'id': $uuid,
>>  'payload': {
>>  'XXX
>>  }
>>  }
>
>
> Thanks, it totally makes sense. So when I convert my example to your usage
> it looks like this:
>
> {
> body: {
> 'timestamp': $timestamp,
> 'type': 'tripleo.validations.v1.run_validation',
> 'id': $uuid,
> 'payload': {
> execution_id: '123123123',
> validation_id: '123321'
> ...
>  }
> }
> }
>
> I am not sure whether to separate the version from type as it would become
> complicated to reconstruct the workflow name (at least for tripleo
> workflows).
> The most important is the 'type' as that is the key which we'd like to use
> on client to identify what action to take.

Looks great to me, thanks!

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Zaqar messages standardization

[Thomas Herve]

On Fri, May 20, 2016 at 5:52 PM, Jiri Tomasek  wrote:
> Hey all,
>
> I've been recently working on getting the TripleO UI integrated with Zaqar,
> so it can receive a messages from Mistral workflows and act upon them
> without having to do various polling hacks.
>
> Since there is currently quite a large amount of new TripleO workflows
> comming to tripleo-common, we need to standardize this communication so
> clients can consume the messages consistently.
>
> I'll try to outline the requirements as I see it to start the discussion.
>
> Zaqar queues:
> To listen to the Zaqar messages it requires the client to connect to Zaqar
> WebSocket, send authenticate message and subscribe to queue(s) which it
> wants to listen to. The currently pending workflow patches which send Zaqar
> messages [1, 2] expect that the queue is created by client and name is
> passed as an input to the workflow [3].
>
> From the client perspective, it would IMHO be better if all workflows sent
> messages to the same queue and provide means to identify itself by carrying
> workflow name and execution id. The reason is, that if client creates a
> queue and triggers the workflow and then disconnects from the Socket (user
> refreshes browser), then it does not know what queues it previously created
> and which it should listen to. If there is single 'tripleo' queue, then all
> clients always know that it is where it will get all the messages from.
>
> Messages identification and content:
> The client should be able to identify message by it's name so it can act
> upon it. The name should probably be relevant to the action or workflow it
> reports on.
>
> {
>   body: {
> name: 'tripleo.validations.v1.run_validation,
> execution_id: '123123123'
> data: {}
>   }
> }
>
> Other parts of the message are optional but it would be good to provide
> information relevant to the message's purpose, so the client can update
> relevant state and does not have to do any additional API calls. So e.g. in
> case of running the validation a message includes validation id.

Hi,

Sorry for not responding earlier, but I have some inputs. In Heat we
publish events on Zaqar queue, and we defined this format:

{
'timestamp': $timestamp,
'version': '0.1',
'type': 'os.heat.event',
'id': $uuid,
'payload': {
'XXX
}
}

I don't think we have strong requirements on that, and we can
certainly make some tweaks. If we can converge towards something
simimar that'd be great.

Thanks,

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] enabled convergence background

[Thomas Herve]

On Fri, May 20, 2016 at 5:46 AM, xiangxinyong  wrote:
> Hi Team,
>
> I noticed that heat enabled convergence.

I hope that's not the case :). We haven't made the switch yet. We
continue to do testing, and we're still finding issues so we won't
make it until we have a good confidence that it's mostly seamless.

> Could someone tell me about the background about the convergence.
> or some specs to introduce it?

I believe the spec is a good introduction to it:
https://specs.openstack.org/openstack/heat-specs/specs/juno/convergence.html

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral][Zaqar] Triggering Mistral workflows from Zaqar messages

[Thomas Herve]

On Wed, May 18, 2016 at 8:49 PM, Zane Bitter  wrote:
> I've been lobbying the Mistral developers for $SUBJECT since, basically,
> forever.[1][2][3] I like to think after a couple of years I succeeded in
> changing their view on it from "crazy" to merely "unrealistic".[4] In the
> last few months I've had a couple of realisations though:
>
> 1) The 'pull' model I've been suggesting is the wrong one, architecturally
> speaking. It's asking Mistral to do too much to poll Zaqar queues.
> 2) A 'push' model is the correct architecture and it already exists in the
> form of Zaqar's Notifications, which suddenly makes this goal look very
> realistic indeed.
>
> I've posted a Zaqar spec for this here:
>
> https://review.openstack.org/#/c/318202/

Commented. I certainly need some more time to think about it.

> Not being super familiar with either project myself, I think this needs
> close scrutiny from Mistral developers as well as Zaqar developers to make
> sure I haven't got any of the details wrong. I'd also welcome any volunteers
> interested in implementing this :)
>
>
> One more long-term thing that I did *not* mention in the spec: there are
> both Zaqar notifications and Mistral actions for sending email and hitting
> webhooks. These are two of the hardest things for a cloud operator to
> secure. It would be highly advantageous if there were only _one_ place in
> OpenStack where these were implemented. Either project would potentially
> work - Zaqar notifications could call a simple, operator defined workflow
> behind the scenes for email/webhook notifications; alternatively the Mistral
> email/webhook actions could drop a message on a Zaqar queue connected to a
> notification - although the former sounds easier to me. (And of course
> clouds with only one of the services available could fall back to the
> current plugins.) Something to think about for the future...

And you're forgetting about Aodh alarm notifications, which are
webhooks as well. Unfortunately, I think none of them do a
particularly great job in terms of robustness and security. I
suggested moving away from doing webhook in Zaqar to Flavio in the
past, and I think he responded that he thought it was part of the core
functionality. It's hard to delegate such operations and create a
dependency on another project. Maybe we can start by creating some
library code.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] Summit recap

[Thomas Herve]

Hi all,

Here are some of my thoughts on the Austin summit sessions. Don't
hesitate to ask questions or fix my misunderstanding. I'll try to keep
it short, please also refer to the etherpads for more details
(sometimes :)):
https://wiki.openstack.org/wiki/Design_Summit/Newton/Etherpads#Heat


Release model
--

We'll continue to as usual in the cycle-with-milestones, and
investigate having a shorter feature freeze for more flexibility.


Default to convergence
-

Now's the time to enable it by default. We have a bunch of small
things to fix in tests, but the main point is to check that projects
using Heat work fine. If we can pull it off I hope to do the switch by
newton-1, so in 3 weeks. That we'll leave us with plenty of time for
feedback and improvements


Convergence improvements


There are some changes that we can now make to use convergence, like
not polling on hooks, which would be nice. But the main part is to
work on performance, to close the gap with the legacy engine. There is
at least one issue with messaging that we need to investigate.


Performance
--

This is what I hope will be a focus for the cycle, especially with
regard to large stacks. We have one major issue with the way we handle
files in the environment, which clogs our memory usage. That fix is in
good progress, once merged we'll find some smaller things to
incrementally improve on. We need to measure more to be able detect
regressions early and test new approaches. I'll see with infra if we
can get a periodic job for that and how storage works.


Continuous observer
--

The outcome of that session was basically that we don't want to do
this in Heat, at least for the time being. First, we're missing the
notifications on the resources that we spawn. Until that functionality
exists in OpenStack, it'll be tough to write a generic solution. Then,
it's a policy driven mechanism, and you need to be able to customize
how to react to failure on resources. So for now we want to document
how you can use the current APIs outside of Heat to achieve this, and
make sure that it works fine.


HOT parser
-

This was somewhat tricky subject, as we tried to mix external template
validation and template building. The former is difficult to pull off
without being wrong, duplicate efforts, or slowing down development
too much. It may be possible to do some basic validation, and we'll
see if we can create a reusable library for that. The latter is a bit
more clear (at least to me), and I hope something will be available
for Newton.


New heatclient commands
--

We listed improvements that can be made now that we completed the move
to OSC. I'm really interested on a way to get failures more easily, as
it's a common pattern and it'll improve user experience quite a bit.


Software deployment refinements


We decided to add a new property on SoftwareConfig indicating the
inputs which creates a replacement. We also need to improve timeouts
management, and allow the ability to send data when deployments are in
progress to get debug information.


Using DLM


We managed to not talk about ZooKeeper too much :). We want to remove
the database locks that we have and use tooz instead. The key is to
provide a nice upgrade path, so starting with the locks use for
convergence seems easier. If it turns out to be painless, we'll move
the stack locks from the legacy engine as well.


Validation improvements
---

Many subjects were mentioned here, with regard to the HOT parser topic
as well. The change I'd like to see is using placeholder objects
everywhere instead of None, so that we can clarify validations of
non-resolved data.


Integration tests
---

Not much to say on that subject. We want to move to use tempest
plugin, and the tempest runner, and remove as much custom code that we
have as possible.


Thanks,

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][heat] Summit session clashes

[Thomas Herve]

On Wed, Apr 20, 2016 at 12:14 PM, Ethan Lynn  wrote:
> Could we move Functional Tests to Thursday? I have a hands-on workshop at
> wed 4:30-6:00 pm.

Sorry, I don't have anything to switch it with.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] No IRC meeting next week

[Thomas Herve]

Hi all,

As expected, no meeting during the summit next week. It will resume
normally the week after, though I won't be around to run it.

Thanks,

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][heat] Summit session clashes

[Thomas Herve]

On Wed, Apr 20, 2016 at 3:26 AM, Zane Bitter  wrote:
> Umm, I thought that was the idea.
>
> I need to be at both of those Heat ones anyway, so this doesn't really help
> me. I'd rather have the DLM session in this slot instead. (The only sessions
> I can really skip are the Release Model, Functional Tests and DLM.) That
> would give us:
>
>   HeatTripleO
>
>  Wed 3:30 Release Model
>  Wed 4:30 HOT Parser
>  Wed 5:20 Functional Tests
>
>  Thu 1:30 DLM Upgrades
>  Thu 2:20 Convergence switchover  Containers
>  Thu 3:10 Convergence cleanup Composable Roles
>  Thu 4:10 Performance API
>  Thu 5:00 Validation  CI
>
> I think that way Steve and I could probably both cover upgrades, and he
> could cover the rest.
>
> I'd like to get to the composable roles and containers sessions too, but
> we'd have to rejig basically every Heat session and I think it's too late to
> be doing that.

Sounds good to me. To be clear, I switched Release model and
Validation improvements, and DLM and Performance.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] Design summit planning

[Thomas Herve]

Hi all,

We've been allocated 12 sessions for Austin. I started an etherpad
where you can put topics that you'd like to talk about:
https://etherpad.openstack.org/p/newton-heat-sessions. Please fill it
up, we'll discuss what we have at the meeting tomorrow. Maybe we'll
wait until next week to finalize the schedule.

Thanks,

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Meeting time

[Thomas Herve]

On Wed, Mar 30, 2016 at 10:26 AM, Thomas Herve <the...@redhat.com> wrote:
> On Wed, Mar 30, 2016 at 10:13 AM, Thomas Herve <the...@redhat.com> wrote:
>> Hi all,
>>
>> I'd like to be able to attend at least one of the IRC meeting, and it
>> doesn't really work right now. So if you'd like to come to meetings,
>> please fill the following doodle:
>> http://doodle.com/poll/4m6aicfnwuug86rs
>
> The times are in UTC!

All right, results are in. It's fairly spread out as expected, but
I'll move the morning one up to 8UTC and the other to 15UTC; We'll try
for a bit and change if necessary.

We're going to fix our alignment with the calendar too, so next
meeting Wednesday at 8UTC.

Thanks!

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] 答复: [Heat] Re-evaluate conditions specification

[Thomas Herve]

On Fri, Apr 1, 2016 at 3:21 AM, Zane Bitter  wrote:
> On 31/03/16 18:10, Zane Bitter wrote:
>>
>>
>> I'm in favour of some sort of variable-based implementation for a few
>> reasons. One is that (5) seems to come up fairly regularly in a complex
>> deployment like TripleO. Another is that Fn::If feels awkward compared
>> to get_variable.
>
>
> I actually have to revise this last part after reviewing the patches.
> get_variable can't replace Fn::If, because we'd still need to handle stuff
> of the form:
>
> some_property: {if: [{get_variable: some_var},
>  {get_resource: res1},
>  {get_resource: res2}]
>
> where the alternatives can't come from a variable because they contain
> resource references and we have said we'd constrain variables to be static.
>
> In fact the intrinsic functions that could be allowed in the first argument
> to the {if: } function would have to constrained in the same way as the
> constraint field in the resource, because we should only validate and obtain
> dependencies from _one_ of the alternates, so we need to be able to
> determine which one statically and not have to wait until the actual value
> is resolved. This is possibly the strongest argument for keeping on the cfn
> implementation course.

We talked about another possibilities on IRC: instead of having a new
section, create a new resource OS::Heat::Value which can hold some
data. It would look like that:

resources:
is_prod:
type: OS::Heat::Value
properties:
value: {equals: {get_param, env}, prod}}

my_resource:
condition: {get_attr: [is_prod, value}}


Sounds like it would be fairly flexible, really easy to implement, and
solve most of the issues I can think of (including being included in
the dependency resolution mechanism for free).

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] 答复: [Heat] Re-evaluate conditions specification

[Thomas Herve]

On Thu, Mar 31, 2016 at 2:25 PM, Huangtianhua  wrote:
> The conditions function has been requested for a long time, and there have 
> been several previous discussions, which all ended up in debating the 
> implementation, and no result.
> https://review.openstack.org/#/c/84468/3/doc/source/template_guide/hot_spec.rst
> https://review.openstack.org/#/c/153771/1/specs/kilo/resource-enabled-meta-property.rst

And for a reason: this is a tricky issue, and introducing imperative
constructs in a template can lead to bad practices.

> I think we should focus on the simplest possible way(same as AWS) to meet the 
> user requirement, and follows the AWS, there is no doubt that we will get a 
> very good compatibility.
> And the patches are good in-progress. I don't want everything back to zero:)
> https://review.openstack.org/#/q/status:open+project:openstack/heat+branch:master+topic:bp/support-conditions-function

I don't say that you should scratch everything. I'm mostly OK with
what has been done, with the exception of the top-level conditions
section. Templates are our user interface, and we need to be very
careful when we introduce new things. 3 years ago following AWS was
the easy path because we didn't have much idea on what to do, but I
believe we now have enough background to be more innovative.

It's also slightly worrying that the spec "only" got 3 cores approving
it, especially on such a touchy subject. I'm guilty as others to not
have voiced my concerns then, though.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Re-evaluate conditions specification

[Thomas Herve]

On Thu, Mar 31, 2016 at 11:39 AM, Qiming Teng
<teng...@linux.vnet.ibm.com> wrote:
> On Thu, Mar 31, 2016 at 10:40:33AM +0200, Thomas Herve wrote:
>> Hi all,
>>
>> As the patches for conditions support are incoming, I've found
>> something in the code (and the spec) I'm not really happy with. We're
>> creating a new top-level section in the template called "conditions"
>> which holds names that can be reused for conditionally creating
>> resource.
>>
>> While it's fine and maps to what AWS does, I think it's a bit
>> short-sighted and limited. What I have suggested in the past is to
>> have a "variables" (or whatever you want to call it) section, where
>> one can declare names and values. Then we can add an intrinsic
>> function to retrieve data from there, and use that for examples for
>> conditions.
>>
>> It solves that particular issue, and it opens some interesting
>> possibilities for reducing duplication in the template, as we could
>> build arbitrary values out of parameters or attributes that can then
>> be reused several times.
>>
>> Thoughts?
>
> Though not very excited by the current approach, I'm still a little bit
> concerned about the extent to which we are extending the 'conditionals'
> support. This is not an easy call given that we don't want HOT to become
> a programming language.
>
> A long time ago, we have a proposal from sdake. That one was stagnated
> for ever. If we want to revisit this topic from step 0, we may as well
> need to define the scope cleanly. For example, are we gonna support some
> idioms for 'if, elif' only or we are introducing 'switch/case' also?
> Are we introducing a boolean evaluator that can be applied to all data
> types so that we get grammar completeness?
> Are we gonna support logical operators as well? e.g. AND, OR, NOT?
> Would this mean we are introducing some "reserved words" to HOT, or some
> more builtin functions?
> If these are to be modeled as functions, do we still want them to be
> implemented at client side, as we do for 'get_file'?

That's an interesting conversation, but not the one I was expecting to
trigger. I believe
http://specs.openstack.org/openstack/heat-specs/specs/mitaka/support-conditions.html
answers most of the questions you're asking. I mostly try to adjust
one point of that spec already approved.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Re-evaluate conditions specification

[Thomas Herve]

On Thu, Mar 31, 2016 at 10:40 AM, Thomas Herve <the...@redhat.com> wrote:
> Hi all,
>
> As the patches for conditions support are incoming, I've found
> something in the code (and the spec) I'm not really happy with. We're
> creating a new top-level section in the template called "conditions"
> which holds names that can be reused for conditionally creating
> resource.
>
> While it's fine and maps to what AWS does, I think it's a bit
> short-sighted and limited. What I have suggested in the past is to
> have a "variables" (or whatever you want to call it) section, where
> one can declare names and values. Then we can add an intrinsic
> function to retrieve data from there, and use that for examples for
> conditions.

I was asked to give examples, here's at least one that can illustrate
what I meant:

parameters:
   host:
  type: string
   port:
  type: string

variables:
   endpoint:
  str_replace:
template:
   http://HOST:PORT/
params:
   HOST: {get_param: host}
   PORT: {get_param: port}

resources:
   config1:
  type: OS::Heat::StructuredConfig
  properties:
config:
   hosts: [{get_variable: endpoint}]

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] Re-evaluate conditions specification

[Thomas Herve]

Hi all,

As the patches for conditions support are incoming, I've found
something in the code (and the spec) I'm not really happy with. We're
creating a new top-level section in the template called "conditions"
which holds names that can be reused for conditionally creating
resource.

While it's fine and maps to what AWS does, I think it's a bit
short-sighted and limited. What I have suggested in the past is to
have a "variables" (or whatever you want to call it) section, where
one can declare names and values. Then we can add an intrinsic
function to retrieve data from there, and use that for examples for
conditions.

It solves that particular issue, and it opens some interesting
possibilities for reducing duplication in the template, as we could
build arbitrary values out of parameters or attributes that can then
be reused several times.

Thoughts?

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Meeting time

[Thomas Herve]

On Wed, Mar 30, 2016 at 10:13 AM, Thomas Herve <the...@redhat.com> wrote:
> Hi all,
>
> I'd like to be able to attend at least one of the IRC meeting, and it
> doesn't really work right now. So if you'd like to come to meetings,
> please fill the following doodle:
> http://doodle.com/poll/4m6aicfnwuug86rs

The times are in UTC!

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] Meeting time

[Thomas Herve]

Hi all,

I'd like to be able to attend at least one of the IRC meeting, and it
doesn't really work right now. So if you'd like to come to meetings,
please fill the following doodle:
http://doodle.com/poll/4m6aicfnwuug86rs

Note that I'll try to make the best choice depending on my selfish
availability, and who's already active currently in meetings. I may
just move the 20UTC meeting to something like 15UTC if that's OK.

Thanks!

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] PTL Candidacy

[Thomas Herve]

Hi everyone,

I'm happy to announce my candidacy for PTL of Heat for the Newton cycle.

The project and the community are in a good place in my opinion currently, both
diverse and active. As much as possible I'd like to continue encourage and
improve that.

Heat is being used more and more by projects inside OpenStack. As we know first
hand, being broken by other projects is not a great experience, so I want to
make sure we don't do this. I believe it goes by being proactive (taking care
of compatibility, making sure gates are not broken before merging) and reactive
(handle issues promptly, not being afraid of reverts).

On the other hand, I also want to reach application deployment outside of
OpenStack itself, with a focus on documentation, and improving our
heat-templates repository.

I don't believe we should work much on particular features outside of
continuing our resource coverage. Pushing convergence, and working on
scalability and performance sounds like what we should aim for in the near
future.

All of that said, being PTL is also a lot about release coordination, which I
hope to learn with the help of our successful lineage of PTLs.

Thanks!

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Nomination Oleksii Chuprykov to Heat core reviewer

[Thomas Herve]

On Wed, Mar 16, 2016 at 11:57 AM, Sergey Kraynev  wrote:
> Hi Heaters,
>
> The Mitaka release is close to finish, so it's good time for reviewing
> results of work.
> One of this results is analyze contribution results for the last release 
> cycle.
> According to the data [1] we have one good candidate for nomination to
> core-review team:
> Oleksii Chuprykov.
> During this release he showed significant value of review metric.
> His review were valuable and useful. Also He has enough level of
> expertise in Heat code.
> So I think he is worthy to join to core-reviewers team.
>
> I ask you to vote and decide his destiny.
>  +1 - if you agree with his candidature
>  -1  - if you disagree with his candidature

+1!

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][zaqar][cloudkitty] Default ports list

[Thomas Herve]

On Thu, Mar 10, 2016 at 2:55 PM, Sean Dague <s...@dague.net> wrote:
> On 03/10/2016 08:40 AM, Thomas Herve wrote:
>> On Thu, Mar 10, 2016 at 2:28 PM, Chris Dent <cdent...@anticdent.org> wrote:
>>> +many. It would be great if we just got rid of the runnable web
>>> servers in the projects and just expose wsgi apps (the tools like
>>> devstack etc mounted under whatever the available server is).
>>
>> Isn't devstack meant for development? Running the APIs in a WSGI
>> container like Apache or uwsgi makes for a terrible debugging
>> experience. Just this morning I had to prevent aodh from running in
>> Apache to be able to run it standalone.
>>
>> Also, those apps that use WSGI still bind a different port. The fact
>> that it runs in Apache doesn't really solve the URLs problem.
>
> But they shouldn't. I do realize it's taking a while to get there, but
> this is the push to get rid of all these weird ports. The point is to
> get them all on 80 in a subpath or a separate domain.
>
>
> If projects use the pbr wsgi_script directive the wsgi script will run
> on wsgi ref on the commandline if you need that for debug -
> https://github.com/openstack/keystone/blob/4db54810e58ad86edb92869a608fb5630a6b99e5/setup.cfg#L75
> that was built to make this simpler.

Right, but if we move to everything in WSGI besides a subpath, you
won't be able to switch to the script easily. Unless you actually
implement that using a reverse proxy.

I totally understand the will to remove those ports from the final
product. I question whether it's the right choice for devstack. It
would seem that at least keeping those 'weird' ports for internal
consumption would be useful. It's not like we're close to use the 65K
available.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][zaqar][cloudkitty] Default ports list

[Thomas Herve]

On Thu, Mar 10, 2016 at 2:28 PM, Chris Dent  wrote:
> On Thu, 10 Mar 2016, Sean Dague wrote:
>
>> These are HTTP services. They really shoudn't be claiming new ports,
>> they should be running on a real webserver on 80 or 443.
>>
>> There is some legacy there with the original services that we are
>> churning through. It would be nice if new services *started* with
>> staying on wsgi only, and stopped grabbing random ports.
>
>
> +many. It would be great if we just got rid of the runnable web
> servers in the projects and just expose wsgi apps (the tools like
> devstack etc mounted under whatever the available server is).

Isn't devstack meant for development? Running the APIs in a WSGI
container like Apache or uwsgi makes for a terrible debugging
experience. Just this morning I had to prevent aodh from running in
Apache to be able to run it standalone.

Also, those apps that use WSGI still bind a different port. The fact
that it runs in Apache doesn't really solve the URLs problem.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] intrinsic function bugfixes and hot versioning

[Thomas Herve]

On Wed, Feb 17, 2016 at 7:54 PM, Steven Hardy  wrote:
> Hi all,
>
> So, Zane and I have discussed $subject and it was suggested I take this to
> the list to reach consensus.
>
> Recently, I've run into a couple of small but inconvenient limitations in
> our intrinsic function implementations, specifically for str_replace and
> repeat, both of which did not behave the way I expected when referencing
> things via get_param/get_attr:

Disclaimer: compatibility is not black and white, especially in these
cases. We need to make decisions based on the impact we can imagine on
users, so it's certainly subjective. That said:

> https://bugs.launchpad.net/heat/+bug/1539737

I think it works fine as a bug fix.

> https://bugs.launchpad.net/heat/+bug/1546684

I agree that a new version would be better.

The main difference for me is that even if it's arguable, you could
build a working template relying on the current behavior (having a
template returned by a function).
If you find a way to keep the current behavior *and* have the one you
expect, then I can see it as a bug fix.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum][heat] Bug 1544227

[Thomas Herve]

On Thu, Feb 11, 2016 at 5:23 PM, Hongbin Lu  wrote:
> Rabi,
>
> As you observed, I have uploaded two testing patches [1][2] that depends on 
> your fix patch [3] and the reverted patch [4] respectively. An observation is 
> that the test "gate-functional-dsvm-magnum-mesos" failed in [1], but passed 
> in [2]. That implies the reverted patch does resolve an issue (although I am 
> not sure exactly how).
>
> I did notice there are several 404 errors from Neutron, but those errors 
> exist in successful tests as well so I don't think they are the root cause.
>
> [1] https://review.openstack.org/#/c/278578/
> [2] https://review.openstack.org/#/c/278778/
> [3] https://review.openstack.org/#/c/278576/
> [4] https://review.openstack.org/#/c/278575/

Hi,

Interestingly, [2] fails with a different error. At some point, we get
the following error:

Unable to find network with name '3bc0ffd2-6c4a-4e46-9a0e-4fbc91920daf'

It doesn't make much sense, because we retrieve that network seconds
before, but suddenly it fails. In the neutron log, you can find this:

SAWarning: The IN-predicate on "ml2_network_segments.network_id" was
invoked with an empty sequence. This results in a contradiction, which
nonetheless can be expensive to evaluate.  Consider alternative
strategies for improved performance.

It's possible that patch highlights a bug in neutron.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Rico Lin for heat-core

[Thomas Herve]

On Mon, Dec 7, 2015 at 1:39 PM, Sergey Kraynev 
wrote:

>
> Hi all.
>
> I'd like to nominate Rico Lin for heat-core. He did awesome job with
> providing useful and valuable reviews. Also his contribution is really high
> [1] .
>
> [1] http://stackalytics.com/report/contribution/heat-group/60
>
> Heat core-team, please vote with:
>  +1 - if you agree
>   -1 - if you disagree
>
>
+1.

-- 
Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] get_reality implementation: default values, specified in config files

[Thomas Herve]

On Thu, Dec 3, 2015 at 1:00 PM, Peter Razumovsky 
wrote:

>
> Hello!
>
> During get_reality implementation next problem raised: if I don't specify
> optional not update allowed property (like availabiliy_zone in
> OS::Cinder::Volume), default value will be set for this property in entity.
> This default value defined in configuration of entity's project (i.e. if I
> create OS::Cinder::Volume without az, volume will be created with az =
> default value, which defined in cinder.conf).
>
> So, if I want to get live state of entity, I get az with value, but in
> template no value has been specified for az, and that's mean, stack will be
> repalced during update, which is awful (e.g. if we have huge stack with
> this problem for each resource, updating will be very slow).
>
> What I can do with such properties with default values, defined in config
> options? I don't know any methods to get value from config of other project.
>

I haven't thought too much about it, but I'm sure getting config from other
projects is not the right idea :). One possible solution would be to get
the live state of the entity just after creation, so that we can fill those
default properties right away. What do you think?

-- 
Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Enabling and Reading notifications

[Thomas Herve]

On Tue, Nov 3, 2015 at 11:22 PM, Pratik Mallya 
wrote:

> Hello,
>
> I was looking for guidance as to how to enable notifications in heat and
> if there is already a tool that can read those events? Looking through the
> code, it gives somewhat conflicting information as to the extent to which
> notifications are supported. e.g. [1] says its not supported, but there is
> an integration test [2] available.
>
>
Hi,

[1] is something totally different, it's the notification property of stack
objects, for AWS compatibility. It was never implemented.

To enable Heat notifications, you simply need to set notification_driver,
messagingv2 being a goind value. It will then be published in your message
bus (depending on how it's configured). Ceilometer will consume those, or
you can plug any tool to read them on the appropriate exchange.

-- 
Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat][Rant] About blank rechecks

[Thomas Herve]

Hi all,

You've seen me complain about people doing blank rechecks in Gerrit on IRC,
and it seems it had little to no effect. So here I am trying to spread the
word here. I'll try to stay calm.

I'm seeing way too many rechecks on heat patches. It's not epidemic, but
it's still enough to make me sad.

First, it makes me sad as a developer. I don't know if it's just me, but
one of the reason I code is curiosity, and debugging a gate failure is a
great way to learn, pierce through the layers, and improve the situation.

It then makes me sad as a team member. By doing a recheck you're basically
implying that you don't care about the failure, and surely someone will
care at some point. Except, the information will be lost, and we may have
100 builds before that happen again, when a release already happened, and
we have to backport it. Working early means working less.

And finally, it makes me sad for the infra team. Doing a recheck is
disrespecting all the work they're doing to create a reliable environment
to run our tests. Sure, sometimes the environment is the reason the failure
happens, but then it's even more important to give feedback about it. They
provide a great deal of logs, we can use logstach to find patterns, the
least we can do is trying. We're also using resources that other projects
could be using. As much as we'd like to believe it, the cloud doesn't have
free infinite resources.

Recently, I've seen many cases where rechecks were made whereas:
1) The heat branch was broken. Generally for some external reason (a
dependency updated), doing a recheck is a pure waste of resources until
that failure is fixed. Most of the time, we say something on IRC when it's
the case. We also try to open a bug, so looking at launchpad can show
something.
2) THE PATCH WAS ACTUALLY BROKEN. And there I'm not sad anymore I'm
particularly angry. It basically means that you didn't look at all at the
build results, and just mindlessly typed rechecks hoping that some fairy
will fix your broken code. Frankly, that makes want to go on a -2 rampage.
ESPECIALLY where a core is doing it.

To close, I'll try to provide a solution. I know we all have our agenda,
debugging gate failures takes some time that you may not have, and
obviously "my patch is fine it's not my fault" (who cares, that's what
being in a team means). Still, I'd like everyone to look at the test
failures, look if the patch is not the problem, and if not open a bug [1]
mentioning the test name, pasting the traceback in the description, and
linking the build result. Then do recheck bug #xyz. That's it. It shouldn't
take you more than 3 minutes, and at least we didn't lose the information.

Thanks for reading that far and sorry for the length,


[1] https://bugs.launchpad.net/heat/+filebug

-- 
Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] core team nomination

[Thomas Herve]

+1!

-- 
Thomas

On Tue, Oct 20, 2015 at 3:38 PM, Sergey Kraynev 
wrote:

> I'd like to propose new candidates for heat core-team:
> Rabi Mishra
> Peter Razumovsky
>
> According statistic both candidate made a big effort in Heat as
> reviewers and as contributors [1][2].
> They were involved in Heat community work  during last several releases and
> showed good understanding of Heat code.
> I think, that they are ready to became core-reviewers.
>
> Heat-cores, please vote with +/- 1.
>
> [1] http://stackalytics.com/report/contribution/heat-group/180
> [2] http://stackalytics.com/?module=heat-group=person-day
> --
> Regards,
> Sergey.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] py.test vs testrepository

[Thomas Herve]

On Wed, Oct 7, 2015 at 12:59 PM, Roman Prykhodchenko  wrote:

> What I can extract now from this thread is that Fuel should switch to
> testr because of the following reasons:
>
> - Diversity of tools is a bad idea on a project scale
> - testrepository and related components are used in OpenStack Infra
> environment for much more tasks than just running tests
> - py.test won’t be added to global-requirements so there always be a
> chance of another dependency hell
> - Sticking to global requirements is an idea which is in the scope of
> discussions around Fuel.
>
> Sounds like that’s the point when we should just file appropriate bugs and
> use testr in smaller components, e. g., Fuel Client, first and then try in
> in Nailgun.
>

I'd say that using testr in the default tox targets and thus in the gate is
the reasonable choice, for the reasons mentioned elsewhere. That said, I
also think that it's also fair to allow alternate test runners on the local
developer environment. You may have to make some tweaks from time to time,
but most of the time py.test should be able to run the test suite supported
by testr (this is not necessarily true for nose for example which doesn't
support testscenarios AFAIK).

This way you standardize on the tools (testr remains the "source of
truth"), but make local debugging much nicer.

-- 
Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack][magnum][heat]problems for synchronizing stack parameters from heat

[Thomas Herve]

 We can get stacks by stack list call, but it does not provide info about
 stack parameters. If we need stack parameters, we have to use stack.get.

Yeah that part is right. I believe we consider stack parameters somewhat 
private to the user, which may be the reason they are not easily accessible. 
What do you need them for?

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack][magnum][heat]problems for synchronizing stack parameters from heat

[Thomas Herve]

 Hi all,
 
 Magnum creates a stack when a bay is created and update the stack
 parameters when the bay is updated. Magnum has a periodic task
 to synchronize stack status from heat.
 And now we want to synchronize  stack parameters from heat, too. But heat
 don't allow admin user to show stack in other tenants, so we can not get
 stack parameters.

That's not true. You just need to pass the appropriate option, global_tenant, 
in your list call.

Regards,

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] status of instance_user/admin_user in heat

[Thomas Herve]

 
 Looks like this change is what I need, but was the code supposed to not set
 it to ec2-user when instance_user is unset? I've always had it unset and
 it's always set it to ec2-user, certainly for Ubuntu images anyway, I've
 not tested others.

It's a common misconception. Unset is different from the empty string. If you 
don't set it, it keeps using the default value, which is ec2-user. If you set 
it to the empty string, it keeps whatever the image is using. If you don't that 
you shouldn't need this change.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] status of instance_user/admin_user in heat

[Thomas Herve]

 Heat devs,
 
 What is the status of the instance_user config option and the corollary
 OS::Nova::Server::admin_user field? Our users find it very confusing when
 Heat creates a VM with a user called ec2-user. While I've told them to
 explicitly set admin_user, the documentation claims its deprecated since
 Icehouse. Will it be removed? Along the same lines, instance_user in the
 config also claims to be deprecated and slated for removal in Kilo, it is
 also still there in master. So what's the right behavior here? Should I
 tell heat users that it's safe/recommended to set admin_user or not?

Hi,

It's been deprecated for a while, and will be hidden in the next release (see 
https://review.openstack.org/#/c/103928/) if everything goes well.

As an operator, you shoud set the instance_user configuration value to the 
empty string, so that your users get the default user from the image. That's 
the recommanded mode of operation.

Cheers,

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] autoscaling and load balancers

[Thomas Herve]

Hi,

Response inline.


 Hi,
 
 The OS::Heat::AutoScalingGroup resource is somewhat limited at this time,
 because when a scaling even occurs it does not notify dependent resources,
 such as a load balancer, that the pool of instances has changed.

That's technically not true. If you use a neutron load balancer, and a 
PoolMember resource in each scaling unit, it will do exactly that.

 The AWS::AutoScaling::AutoScalingGroup resource, on the other side, has a
 LoadBalancerNames property that takes a list of
 AWS::ElasticLoadBalancing::LoadBalancer resources that get updated anytime
 the size of the ASG changes.
 
 I'm trying to implement this notification mechanism for HOT templates, but
 there are a few aspects that I hope to do better.
 
 1. A HOT template can have get_attr function calls that invoke attributes of
 the ASG. None of these update when the ASG resizes at this time, a scaling
 even does a partial update that only affects the ASG resource. I would like
 to address this.
 
 2. The AWS solution relies on the well known LoadBalancer resource, but often
 load balancers are just regular instances that get loaded with a load
 balancer such as haproxy in a custom way. I'd like custom load balancers to
 also update when the ASG resizes.
 
 The ResourceGroup is an interesting resource. It is much simpler than the
 ASG. In particular, the only way to scale the ResourceGroup is by issuing a
 stack-update with a new size. This indirectly solves #1 and #2 above,
 because when a full update is issued any references to the ResourceGroup get
 updated as well.
 
 In my opinion, the best way to address #1 and #2 above so that they work for
 the ASG as they work for the RG, is to change what happens when there is a
 scaling event. When the ScalingPolicy resource gets a signal, it reaches
 directly to the ASG by calling asg.adjust() (or in the near future by
 sending a signal to it, when a currently proposed patch merges) with the new
 size. This bypasses the update mechanism, so only a partial update occurs,
 just the ASG resource itself is updated. I would like this to be a full
 stack update, so that all references get updated with the new ASG size. This
 will address #1 and #2.

I'm in full support of this. We already do a localized stack update on the 
nested stack, I don't see any reason why we would do a full one. It would make 
the template works without making the user do any extra declaration.

 But there is an alternative to this. I guess we could copy the update
 mechanism used on the AWS side, which is also partial, but at least covers
 the load balancers, given in the LoadBalancerNames property. We can have a
 load_balancer_names equivalent property for the OS::Heat::ASG resource,
 and we can then trigger the updates of the load balancer(s) exactly like the
 AWS side does it. For this option, I would like to extend the load balancer
 update mechanism to work on custom load balancers, as it currently works
 with the well known load balancer resources. I have implemented this
 approach and is currently up for review:
 https://review.openstack.org/#/c/170634/ . I honestly prefer the full
 update, seems cleaner to me.

As mentioned in the review, I don't think it's the proper approach. Especially 
considering that the load_balancer_names property would actually contain 
arbitrary resources.

 Anyway, sorry for the long email. If you can provide guidance on which of the
 approaches are preferred, or if you have other ideas, I would appreciate it.

Thanks a lot for the careful explanation and for tackling this issue.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Barbican : Unable to reterieve asymmetric order request for rsa algorithm type

[Thomas Herve]

 Hi All ,
 
 Could any one please let me know if Barbican would support asymmetric order
 retrieval and the request .
 Please find the curl command and response for creating the order and
 retrieving the order
 
 root@barbican:~# curl -X POST -H 'content-type:application/json' -H
 'X-Project-Id: 12345' -d '{type : asymmetric, meta: {name:
 secretnamepk2, algorithm: rsa, bit_length: 256, mode: cbc,
 payload_content_type: application/octet-stream}}'
 http://localhost:9311/v1/orders
 {order_ref: 
 http://localhost:9311/v1/orders/f9870bb5-4ba3-4b19-9fe3-bb0c2a53557c
 }root@barbican:~#
 
 root@barbican:~# curl -H 'Accept: application/json' -H 'X-Project-Id:12345'
 http://localhost:9311/v1/orders/f9870bb5-4ba3-4b19-9fe3-bb0c2a53557c
 {status: ERROR, updated: 2015-03-30T21:36:38.102832, created:
 2015-03-30T21:36:38.083428, order_ref: 
 http://localhost:9311/v1/orders/f9870bb5-4ba3-4b19-9fe3-bb0c2a53557c ,
 meta: {name: secretnamepk2, algorithm: rsa,
 payload_content_type: application/octet-stream, mode: cbc,
 bit_length: 256, expiration: null},  error_status_code: 400,
 error_reason: Process TypeOrder issue seen - No plugin was found that
 could support your request., type: asymmetric} root@barbican:~#
 
 It seems that we are unable to reterive the asymmetric order request
 Could any one please help.
 Any Help would be appreciated.

Hi,

I don't know about your plugins, but I believe the default one needs at least 
1024 of bit length.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Block migrations and Cinder volumes

[Thomas Herve]

 Interesting bug.  I think I agree with you that there isn't a good solution
 currently for instances that have a mix of shared and not-shared storage.
 
 I'm curious what Daniel meant by saying that marking the disk shareable is
 not
 as reliable as we would want.

I think this is the bug I reported here: 
https://bugs.launchpad.net/nova/+bug/1376615

My initial approach was indeed to mark the disks are shareable: the patch 
(https://review.openstack.org/#/c/125616/) has comments around the issues, 
mainly around I/Ocache and SELinux isolation being disabled.

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] if by archived you mean, wipes out your tables completely, then sure, it works fine

[Thomas Herve]

[snip]
 If we assume that all of our tables are filled up with zeroes for those
 deleted columns, because that’s the default, this **wipes the whole table
 clean**.
 
 How do the tests pass? Well the tests are in test_db_api-ArchiveTestCase,
 and actually, they don’t. But they don’t fail every time, because the test
 suite here runs with a database that is almost completely empty anyway, so
 the broken archival routine doesn’t find too many rows to blow away except
 for the rows in “instance_types”, which it only finds sometimes because the
 tests are only running it with a small number of things to delete and the
 order of the tables is non-deterministic.
 
 I’ve posted the bug report at https://bugs.launchpad.net/nova/+bug/1431571
 where I started out not knowing much about how this worked except that my
 tests were failing, and slowly stumbled my way to come to this conclusion. A
 patch is at https://review.openstack.org/#/c/164009/, where we look at the
 actual Python-side default. However I’d recommend that we just hardcode the
 zero here, since that’s how our soft-delete columns work.

Hi Mike,

Thanks for the investigation. I was wondering when that behavior was introduced 
and it seems that 
http://git.openstack.org/cgit/openstack/nova/commit/?id=ecf74d4c0a5a8a4290ecac048fc437dafe3d40fc
 is the likely culprit, which would mean that only Kilo is affected. Can you 
confirm?

Thanks,

-- 
Thomas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] core team changes

[Thomas Herve]

 Hi all
 
 After having a look at the stats:
 http://stackalytics.com/report/contribution/heat-group/90
 http://stackalytics.com/?module=heat-groupmetric=person-day
 
 I'd like to propose the following changes to the Heat core team:
 
 Add:
 Qiming Teng
 Huang Tianhua
 
 Remove:
 Bartosz Górski (Bartosz has indicated that he is happy to be removed and
 doesn't have the time to work on heat ATM).
 
 Core team please respond with +/- 1.
 
 Thanks
 Angus

+1!

-- 
Thomas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Automatic evacuate

[Thomas Herve]

  This still doesn't do away with the requirement to reliably detect
  node failure, and to fence misbehaving nodes. Detecting that a node
  has failed, and fencing it if unsure, is a prerequisite for any
  recovery action. So you need Corosync/Pacemaker anyway.
 
  Obviously, yes.  My post covered all of that directly ... the tagging
  bit was just additional input into the recovery operation.
 
 This is essentially why I am saying using the Pacemaker stack is the
 smarter approach than hacking something into Ceilometer and Heat. You
 already need Pacemaker for service availability (and all major vendors
 have adopted it for that purpose), so a highly available cloud that
 does *not* use Pacemaker at all won't be a vendor supported option for
 some time. So people will already be running Pacemaker — then why not
 use it for what it's good at?

I may be missing something, but Pacemaker will only provide monitoring of your 
compute node, right? I think the advantage you would get by using something 
like Heat is having an instance agent and provide monitoring of your client 
service, instead of just knowing the status of your hypervisor. Hosts can fail, 
but there is another array of failures that you can't handle with the global 
deployment monitoring.

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Nominating Pavlo Shchelokovskyy for heat-core

[Thomas Herve]

 I'd like to propose that we add Pavlo Shchelokovskyy to the heat-core team.
 
 Pavlo has been a consistently active member of the Heat community - he's
 a regular participant in IRC and at meetings, has been making plenty of
 good commits[1] and maintains a very respectable review rate[2] with
 helpful comments. I think he's built up enough experience with the code
 base to be ready for core.
 
 Heat-cores, please vote by replying to this thread. As a reminder of
 your options[3], +1 votes from 5 cores is sufficient; a -1 is treated as
 a veto.

+1.

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Introducing Re-Heat

[Thomas Herve]

 
 
 Hello All,
 
 
 
 My name is Mika Ayenson and I have to privilege to intern at Johns Hopkins -
 Applied Physics Lab. I’m really excited to release the latest proof of
 concept “Re-Heat” Re-Heat is a JHUAPL developed tool for OpenStack users to
 help them quickly rebuild their OpenStack environments via OpenStack’s Heat
 .
 
 
 
 Here is a link to the Re-Heat paper:
 https://drive.google.com/open?id=0BzTq-ZB9F-b9b0ZXdy1PT2t3dk0authuser=0
 
 Here is a link to Re-Heat: https://github.com/Mikaayenson/ReHeat

Hi Mika,

It looks very similar to the flame project I've been contributing recently: 
https://github.com/cloudwatt/flame

It's be interesting to see if we can join forces.

Cheers,

--
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] health maintenance in autoscaling groups

[Thomas Herve]

  The check url is already a part of Neutron LBaaS IIRC.
 
 Yep. LBaaS is a work in progress, right?

You mean more than OpenStack in general? :) The LBaaS API in Neutron has been 
working fine since Havana. It's certainly has shortcomings and it seems there 
is a big refactoring in plan, though.

 Those of use using Nova networking are not feeling the love, unfortunately.

That's to be expected. nova-network is going to be supported, but you won't get 
new features for it.

 As far as Heat goes, there is no LBaaS resource type. The
 OS::Neutron::LoadBalancer resource type does not have any health checking
 properties.

There are 4 resources related to neutron load balancing. 
OS::Neutron::LoadBalancer is probably the least useful and the one you can 
*not* use, as it's only there for compatibility with 
AWS::AutoScaling::AutoScalingGroup. OS::Neutron::HealthMonitor does the health 
checking part, although maybe not in the way you want it.

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] [Heat] Ceilometer aware people, please advise us on processing notifications..

[Thomas Herve]

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 On 24/06/14 09:28, Clint Byrum wrote:
  Hello! I would like to turn your attention to this specification draft
  that I've written:
  
  https://review.openstack.org/#/c/100012/1/specs/convergence-continuous-observer.rst
  
  Angus has suggested that perhaps Ceilometer is a better place to handle
  this. Can you please comment on the review, or can we have a brief
  mailing list discussion about how best to filter notifications?
  
  Basically in Heat when a user boots an instance, we would like to act as
  soon as it is active, and not have to poll the nova API to know when
  that is. Angus has suggested that perhaps we can just tell ceilometer to
  hit Heat with a web hook when that happens.
 
 We could have a per-resource webhook (either the signal or something like it)
 that is just a logical notification/kick to go and re-check the resource
 state.
 
 The other part of this is when we turn on continuous convergence, we could
 get an alarm whenever that resource changes state (or what ever we are
 interested
 in - as long as it is in the notification payload).
 
 Given the number of resources we want to manage the alarm sub-system will
 need to
 be scalable. I'd rather Ceilometer solve that than Heat.

When we talked about that issue in Atlanta, I think we came to the conclusion 
that one system wouldn't solve it, and that we need to be able to provide 
different pluggable solutions.

The first solution is just to move the current polling system and create a 
generic API around it. That's something that we'll want to keep, even if it's 
only to make standalone mode works. The next solution for me is to subscribe 
directly to the notification system. We know the shortcomings, but it's the 
obvious improvement we can do in the Juno cycle.

Later on, if/when Ceilometer provides what we need, we can implement a new 
backend using it.

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Import errors in tests are not reported in python 2.7

[Thomas Herve]

Hi all,

I don't know if it's a know issue or not, but I noticed on one of my patch 
(https://review.openstack.org/#/c/99648/) that the 2.7 gate was passing whereas 
the 2.6 is failing because of import errors. It seems to be a problem related 
to the difference in the discover module, so presumably an issue in testtools. 
The problem appears locally when using tox, so it's fairly annoying not to be 
able to trust the test result.

Thanks,

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Import errors in tests are not reported in python 2.7

[Thomas Herve]

 On 06/12/2014 10:32 AM, Thomas Herve wrote:
  Hi all,
  
  I don't know if it's a know issue or not, but I noticed on one of my patch
  (https://review.openstack.org/#/c/99648/) that the 2.7 gate was passing
  whereas the 2.6 is failing because of import errors. It seems to be a
  problem related to the difference in the discover module, so presumably an
  issue in testtools. The problem appears locally when using tox, so it's
  fairly annoying not to be able to trust the test result.
  
  Thanks,
  
 
 I'm actually really confused by the testr commands that are generated
 here -
 http://logs.openstack.org/48/99648/2/check/gate-heat-python27/aa8fead/console.html
 
 vs. what you see in a Nova unit test run.
 
 Is there some particular wrappers you have kicking off in the unit tests?

Hum, not that I know. Our tox.ini does python setup.py testr --slowest

--
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Proposing Thomas Spatzier for heat-core

[Thomas Herve]

 Resending with [Heat] in the subject line. My bad.
 
 On 22/04/14 14:21, Zane Bitter wrote:
  I'd like to propose that we add Thomas Spatzier to the heat-core team.
 
  Thomas has been involved in and consistently contributing to the Heat
  community for around a year, since the time of the Havana design summit.
  His code reviews are of extremely high quality IMO, and he has been
  reviewing at a rate consistent with a member of the core team[1].
 
  One thing worth addressing is that Thomas has only recently started
  expanding the focus of his reviews from HOT-related changes out into the
  rest of the code base. I don't see this as an obstacle - nobody is
  familiar with *all* of the code, and we trust core reviewers to know
  when we are qualified to give +2 and when we should limit ourselves to
  +1 - and as far as I know nobody else is bothered either. However, if
  you have strong feelings on this subject nobody will take it personally
  if you speak up :)
 
  Heat Core team members, please vote on this thread. A quick reminder of
  your options[2]:
  +1  - five of these are sufficient for acceptance
0  - abstention is always an option
  -1  - this acts as a veto
 
  cheers,
  Zane.

+1!

That's the beginning of the Thomas' versus the Steves.


-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] Stack snapshots

[Thomas Herve]

Hi all,

I started working on the stack snapshot blueprint [1] and wrote a first series 
of patches [2] to get a feeling of what's possible. I have a couple of related 
design questions though:

 * Is a stack snapshot independent of the stack? That's the way I chose for my 
patches, you start with a stack, but then you can show and delete snapshots 
independently. The main impact will be how restoration works: is restoration an 
update action on a stack towards a specific state, or a creation action with 
backup data?

 * Consequently, should be use volume backups (which survive deleting of the 
original volumes) or volume snapshots (which don't). If the snapshot is 
dependent of the stack, then we can use the more efficient snapshot operation. 
But backup is also an interesting use-case, so should it be another action 
completely?


[1] https://blueprints.launchpad.net/heat/+spec/stack-snapshot

[2] 
https://review.openstack.org/#/q/status:open+project:openstack/heat+branch:master+topic:bp/stack-snapshot,n,z

Thanks,

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] more oslo liaisons needed

[Thomas Herve]

 I see that several projects have their Oslo liaisons lined up
 (https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons). It would be
 great if we had at least one volunteer from each project before the
 summit, so we can get a head start on the coordination work.

Hi Doug,

I'd be happy to handle the role for Heat.

Cheers,

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Managing changes to the Hot Specification (hot_spec.rst)

[Thomas Herve]

 Hi folks,
 
 There are two problems we should address regarding the growth and change
 to the HOT specification.
 
 First our +2/+A process for normal changes doesn't totally make sense
 for hot_spec.rst.  We generally have some informal bar for controversial
 changes (of which changes to hot_spec.rst is generally considered:).  I
 would suggest raising the bar on hot_spec.rst to at-least what is
 required for a heat-core team addition (currently 5 approval votes).
 This gives folks plenty of time to review and make sure the heat core
 team is committed to the changes, rather then a very small 2 member
 subset.  Of course a -2 vote from any heat-core would terminate the
 review as usual.
 
 Second, There is a window where we say hey we want this sweet new
 functionality yet it remains unimplemented.  I suggest we create some
 special tag for these intrinsics/sections/features, so folks know they
 are unimplemented and NOT officially part of the specification until
 that is the case.
 
 We can call this tag something simple like
 *standardization_pending_implementation* for each section which is
 unimplemented.  A review which proposes this semantic is here:
 https://review.openstack.org/85610
 
 My goal is not to add more review work to people's time, but I really
 believe any changes to the HOT specification have a profound impact on
 all things Heat, and we should take special care when considering these
 changes.
 
 Thoughts or concerns?

Hi Steve, 

I'm -1 on merging the docs. Regardless of the warnings we put, people are going 
to be confused seeing features here that they can't use. There is also a huge 
changes that the implementation will change from the original doc, thus making 
us forced to update it if/when we merge.

AFAIK gerrit is persistent, so we can keep the doc patch in it forever and link 
it in a blueprint. And merge the doc change alongside the implementation.

Cheers,

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Some thoughts on the mapping section

[Thomas Herve]

 Hi,

Hi Qiming
 
 Regarding the discussion about the adding of a native 'mappings' section
 to HOT [1], which has been abandoned, I have some different thoughts for
 the team to consider.  Maybe having that section added is not a bad
 idea.
 
 I do think we need a 'mappings' kind of functionality in HOT, to allow
 users to make a from available options.  There are objections to adopt
 this change:
 
   1) Too similar to CFN... Which should not be the criteria for
   excluding this.  We can use other section names, if we really believe
   that is an important factor to consider.

I don't think that argument has been used. What Thomas said is that CFN having 
this feature is not a reason for having it in Heat.

   2) The functionality can be covered by 'environments'.  That is
  where the team really concerns about.
 
 My understanding to the item 2) is that people might have been somehow
 misled by the existing sample templates or the examples used in
 discussion (most of which were using imageId as an example).
 
 A 'mappings' section, IMO, provides the user an opportunity to make
 selections from many options, period.  Its usage is far beyond guest
 image Id selection. Not all use cases are suitable to be expressed as
 'environments'.
 
 As the name suggests, 'environment' is a concept to decouple a
 template from the cloud/host/guest where it is created.  It helps
 promote portability or reusability of the template.  If you have
 some settings that will change between clouds, or different guest
 OS. 'environment' is the ideal place to put those variances.
 
 However, a 'mappings' section is about functionality, not usage
 scenario.  It can be treated as a subsection of 'parameters', to
 make a template even more reusable when the stack can be used with
 different resources (say, softwareconfigs, cloudconfigs, different
 sources for get_file etc.).
 
 Say I have three DB backends to choose from, for my application [2].
 The DB backend is better treated as a component of the application,
 instead of part of the environment.  A 'mappings' section can be
 used for a quick selection of OpenStack level options (such as
 scaling policies, security groups, different templates for a nested
 stack, different versions of external templates residing on a GIT
 website...) or any application-level options (such as a different
 SoftwareDeployment for the same SoftwareConfig), irrespective to which
 cloud (the 'environments', my understanding) is hosting the stack.

Environments are not just a way to customize depending on the target cloud. 
They allow you to define values which are going to be available during your 
template instantiation, even on the same cloud. I see more them as complex 
parameters.

I think there was a push back on the review you mentioned because the use case 
is not super clear, and that using mapping for it clearly reduce reusability 
and portability. And the usecase you mentioned is *also* solved by 
environments. So currently you may understand where we're at.

 Speaking of offering options for selection, there is another proposal on
 adding conditional creation of resources [3], whose use case to enable
 or disable a resource creation (among others).  My perception is that
 these are all relevant enhancements to the reusability of HOT templates,
 though I don't think we really need very sophisticated combinatory
 conditionals.

I think that's interesting that you mentioned that, because Zane talked about a 
variables section, which would encompass what conditions and mappings 
mean. That's why we're discussing extensively about those design points, to see 
where we can be a bit more generic to handle more use cases.

Cheers,

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] PTL Candidacy

[Thomas Herve]

Hello all,

I'd like announce my candidacy for Heat PTL.

It's been a pleasure to be part of such a great team for about a year now, and 
that's the main driver for me to want to represent all of you. I trust that as 
a team we'll continue do a good job delivering nice software, and I hope I 
could become the team delegate for the next 6 months.

If I had to choose my slogan for this period, it would be Less features, more 
users. I think we've reached the point where Heat fills a solid portion of the 
use cases it aimed at the beginning, so we should focus on everything else:
 * It starts by more functional testing, via more tempests and grenade 
integration to make sure upgrades are fine. I hope it'll continue to improve 
our stability
 * Then I'd like to look at performance, to make sure we're good enough so that 
people doesn't notice Heat when deploying it
 * We should also maintain our efforts on documentation
 * Finally, we should support template authors, by talking to the OpenStack 
community (TripleO, Trove, Murano,...) but also outside to application 
developers. I hope we can push contributions to heat-templates (or another 
repository) for better community interaction.

I'm willing to learn if I'll actually have some time to do more than one of 
those things while holding the position!

My commits: https://review.openstack.org/#/q/owner:therve,n,z

And my reviews: https://review.openstack.org/#/q/reviewer:therve,n,z

Thanks,

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Murano][Heat] MuranoPL questions?

[Thomas Herve]

 
 What I can say is that I'm not convinced. The only use-case for a DSL would
 be if you have to upload user-written code, but what you mentioned is a Web
 interface, where the user doesn't use the DSL, and the cloud provider is the
 developer. There is no reason in this case to have a secure environment for
 the code.
 
 I didn't say that. There are at least 2 different roles application
 developers/publishers and application users. Application developer is not
 necessary cloud provider. The whole point of AppCatalog is to support
 scenario when anyone can create and package some application and that
 package can be uploaded by user alone. Think Apple AppStore or Google Play.
 Some cloud providers may configure ACLs so that user be allowed to consume
 applications they decided while others may permit to upload applications to
 some configurable scope (e.g. apps that would be visible to all cloud users,
 to particular tenant or be private to the user). We also think to have some
 of peer relations so that it would be possible to have application upload in
 one catalog to become automatically available in all connected catalogs.
 
 This is similar to how Linux software repos work - AppCatalog is repo, Murano
 package is what DEB/RPMs are to repo and DSL is what DEB/RPMs manifests are
 to packages. Just that is run on cloud and designed to handle complex
 multi-node apps as well as trivial ones in which case this may be narrowed
 to actual installation of DEB/RPM

I'm glad that you bring packages up. This is a really good example of why you 
don't need a new programming language. Packages uses whatever technology they 
prefer to handle their scripting needs. They then have an declarative interface 
which hides the imperative parts behind.

You trust OpenStack developers with their code, you trust package developers 
with their code, why not trust catalog developers?

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Murano][Heat] MuranoPL questions?

[Thomas Herve]

 Hi Thomas,
 
 I think we went to the second loop of the discussion about generic language
 concepts. Murano does not use a new language for the sole purpose of having
 parameters, constraints and polymorphism. These are generic concepts which
 are common for different languages, so keeping arguing about these generic
 concepts is just like a holy war like Python vs. C. Keeping these arguments
 is just like to say that we don't need Python as functions and parameters
 already exists in C which is used under the hood in Python.
 
 Yes Murano DSL have some generic concepts similar to HOT. I think this is a
 benefit as user will see the familiar syntax constructions and it will be a
 lower threshold for him to start using Murano DSL.
 
 In a simplified view Murano uses DSL for application definition to solve
 several particular problems:
 a) control UI rendering of Application Catalog
 b) control HOT template generation
 
 These aspects are not covered in HOT and probably should not be covered. I
 don't like an idea of expressing HOT template generation in HOT as it sounds
 like a creation another Lisp like language :-)

I'm not saying that HOT will cover all your needs. I think it will cover a 
really good portion. And I'm saying that for the remaining part, you can use an 
existing language and not create a new one.

 I don't think that your statement that most of the people in the community
 are against new DSL is a right summary. There are some disagreements how it
 should look like and what are the goals. You will be probably surprised but
 we are not the first who use DSL for HOT templates generation. Here is an
 e-mail thread right about Ruby based DSL used in IBM for the same purpose:
 http://lists.openstack.org/pipermail/openstack-dev/2014-February/026606.html
 
 The term Orchestration is quite generic. Saying that orchestration should
 be Heat job sounds like a well know Henry Ford's phrase You can have any
 colour as long as it's black..

That worked okay for him :).

 I think this is again a lack of understanding of the difference between
 Orchestration program and Heat project. There are many aspects of
 Orchestration and OpenStack has the Orchestration program for the projects
 which are focused on some aspects of orchestration. Heat is one of the
 project inside Orchestration program but it does not mean that Heat should
 cover everything. That is why we discussed in this thread how workflows
 aspects should be aligned and how they should be placed into this
 Orchestration program.

Well, today Heat is the one and only program in the Orchestration program. If 
and when you have orchestration needs not covered, we are there to make sure 
Heat is not the best place to handle them. The answer will probably not Heat 
forever, but we need good use cases to delegate those needs to another project.


-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Murano][Heat] MuranoPL questions?

[Thomas Herve]

Hi Stan,

Comments inline.

 Zane,
 
 I appreciate your explanations on Heat/HOT. This really makes sense.
 I didn't mean to say that MuranoPL is better for Heat. Actually HOT is good
 for Heat's mission. I completely acknowledge it.
 I've tried to avoid comparison between languages and I'm sorry if it felt
 that way. This is not productive as I don't offer you to replace HOT with
 MuranoPL (although I believe that certain elements of MuranoPL syntax can be
 contributed to HOT and be valuable addition there). Also people tend to
 protect what they have developed and invested into and to be fair this is
 what we did in this thread to great extent.
 
 What I'm trying to achieve is that you and the rest of Heat team understand
 why it was designed the way it is. I don't feel that Murano can become
 full-fledged member of OpenStack ecosystem without a bless from Heat team.
 And it would be even better if we agree on certain design, join our efforts
 and contribute to each other for sake of Orchestration program.

Note that I feel that most people outside of the Murano project are against the 
idea of using a DSL. You should feel that it could block the integration in 
OpenStack.

 I'm sorry for long mail texts written in not-so-good English and appreciate
 you patience reading and answering them.
 
 Having said that let me step backward and explain our design decisions.
 
 Cloud administrators are usually technical guys that are capable of learning
 HOT and writing YAML templates. They know exact configuration of their cloud
 (what services are available, what is the version of OpenStack cloud is
 running) and generally understands how OpenStack works. They also know about
 software they intent to install. If such guy wants to install Drupal he
 knows exactly that he needs HOT template describing Fedora VM with Apache +
 PHP + MySQL + Drupal itself. It is not a problem for him to write such HOT
 template.
 
 Note that such template would be designed for very particular configuration.
 There are hundreds of combinations that may be used to install that Drupal -
 use RHEL/Windows/etc instead of Fedora, use ngnix/IIS/etc instead of Apache,
 use FastCGI instead of mod_php, PostgreSQL instead of MySQL. You may choose
 to have all software on single VM or have one VM for database and another
 for Drupal. There are also constraints to those combinations. For example
 you cannot have Fedora + IIS on the same VM. You cannot have Apache and
 Drupal on different VMs.
 
 So the HOT template represent fixed combination of those software components.
 HOT may have input parameters like username or dbImageName but the
 overall structure of template is fixed. You cannot have template that choses
 whether to use Windows or Linux based on parameter value. You cannot write
 HOT that accepts number of instances it allowed to create and then decide
 what would be installed on each of them. This is just not needed for Heat
 users.
 
 With Murano the picture looks the opposite. Typical Murano user is a guy who
 bought account from cloud hosting vendor (cloud operator) and want to run
 some software in the cloud. He may not even be aware that it is OpenStack.
 He knows nothing about programming in general and Heat in particular. He
 doesn't want to write YAMLs. He may not know how exactly Drupal is installed
 and what components it consists of.

So that's where I want to make a first stop. If your primary user is not a 
developer, there is no reason to introduce a DSL for security reasons. The 
provider can trust the code he writes, and there is no need to create a 
dedicated language.

 So what he does is he goes to his cloud (Murano) dashboard, browses through
 application catalog, finds Drupal and drags it onto his environment board
 (think like Visio-style designer). He can stop at this point, click deploy
 button and the system will deploy Drupal. In another words the system (or
 maybe better to say cloud operator or application developer) decides what
 set of components is going to be installed (like 1 Fedora VM for MySQL and 1
 CentOS VM for Apache-PHP-Drupal). But user may decide he wants to customize
 his environment. He digs down and sees that Drupal requires database
 instance and the default is MySQL. He clicks on a button to see what are
 other options available for that role.
 
 In Heat HOT developer is the user. But in Murano those are completely
 different roles. There are developers that write application definitions
 (that is DSL code) and there are end users who compose environments from
 those applications (components). Application developers may have nothing to
 do with particular cloud their application deployed on. As for Drupal
 application the developer knows that Drupal can be run with MySQL or
 PostgreSQL. But there may be many compatible implementations of those DBMSes
 - Galera MySQL, TroveMySQL, MMM MySQL etc. So to get a list of what
 components can be placed in a database role Murano needs to look at 

Re: [openstack-dev] [heat][neutron] OS::Heat::AutoScalingGroup and OS::Neutron::PoolMember?

[Thomas Herve]

 Has anybody exercised the case of OS::Heat::AutoScalingGroup scaling a nested
 stack that includes a OS::Neutron::PoolMember? Should I expect this to work?

Hi Mike,

Yes I tested it and it works. I'm trying to build an example for heat-templates 
putting it all together. I'm mostly struggling with how to include the nested 
stack nicely (without a HTTP URL), but everything else seems fine.

Cheers,

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] lazy translation is breaking Heat

[Thomas Herve]

 This change was recently merged:
 
 https://review.openstack.org/#/c/69133/
 
 Unfortunately it didn't enable lazy translations for the unit tests, so it
 didn't catch the many places in Heat that won't work when lazy translations
 are enabled. Notably there are a lot of cases where the code adds the result
 of a call to _() with another string, and Message objects (which are
 returned from _ when lazy translations are enabled) can't be added,
 resulting in an exception being raised.
 
 I think the best course of action would be to revert this change, and then
 reintroduce it along with patches to fix all the problems, while enabling it
 for the unit tests so bugs won't be reintroduced in the future.

+1 for me to revert it. It broke Heat before, and it did it again because we 
didn't have any tests. Let's have a proper test environment for not making that 
mistake again and again.

-- 
Thomas

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Nominate Jason Dunsmore for heat-core

[Thomas Herve]

+1!

-- 
Thomas

- Mail original -
 On Mon, Feb 10, 2014 at 11:38:40AM +1300, Steve Baker wrote:
  I would like to nominate Jason Dunsmore for heat-core.
  
  His reviews are valuable and prolific, his code contributions have
  demonstrated a good knowledge of heat internals, and he has endured a
  sound hazing to get multi-engine into heat.
  
  http://russellbryant.net/openstack-stats/heat-reviewers-60.txt
  http://www.stackalytics.com/?release=icehousemetric=commitsuser_id=jasondunsmore
  
  Heat cores, please reply with your vote.
 
 +1, and thanks for sticking with the multi-engine work, I know it's been a
 tough challenge at times! :)
 
 Steve
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev