[openstack-dev] [Designate] In what sense is it multi-tenant?
In what sense is Designate multi-tenant? Can it be programmed to give different views to different DNS clients? (If so, how?) Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [designate] Status of the project
Joshua Harlowwrote on 02/10/2017 12:21:08 PM: > Knowing where this is at and the issues. It makes me wonder if it is > worthwhile to start thinking about how we can start to look at 'outside > the openstack' projects for DNS. I believe there is a few that are > similar enough to designate (though I don't know well enough) for > example things like SkyDNS (or others which I believe there are a few). > > Perhaps we need to start thinking outside the openstack 'box' in regards > to NIH syndrome and accept the fact that we as a community may not be > able to recreate the world successfully in all cases (the same could be > said about things like k8s and others). > > If we got out of the mindset of openstack as a thing must have tightly > integrated components (over all else) and started thinking about how we > can be much more loosely coupled (and even say integrating non-python, > non-openstack projects) would that be beneficial (I think it would)? I think you might be on to something. The Kubernetes community seems to be thinking about an external DNS service too. I see https://github.com/kubernetes-incubator/external-dns was just created, but do not know anything more about it. Regards, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Kuryr] [Neutron] Controlling security groups through Kuryr's libnetwork plugin [was: Waiting until Neutron PortisActive]
Antoni Segura Puimedonwrote on 06/11/2016 07:39:41 PM: > Well, with a label you can make the Neutron Port have an SG that > forbids pinging. Wait, what? Labels on what can do what? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron PortisActive
What about pinging? BTW, from where would the pings come? In the Docker/Swarm API today there is no way to disable ping. However, once Kuryr's libnetwork plugin is updated so that `docker network connect --ip=W.X.Y.Z ...` will latch onto a matching pre-existing Neutron Port, if it exists, there will be a way for a user to disable pings (right?). In the Kubernetes API there is now a way to do something like security groups, it is called NetworkPolicy; it is not yet well defined enough to say whether it gives the user a way to disable pings. Thanks, Mike From: Mohammad Banikazemi/Watson/IBM@IBMUS To: "OpenStack Development Mailing List \(not for usage questions\)"Date: 06/10/2016 10:50 AM Subject:Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron PortisActive Hi Neil, Currently, when a docker libnetwork "join" operation in Kuryr is returned, it is not guaranteed that the network connectivity has been established. There are containers that check for network connectivity as the first thing they do when they come up and under heavy load some notice there is no connectivity and simply bail out. I am trying to deal with such a use case, Thanks for pointing out that option 2 won't work for you. I think Salvatore also alluded to that in his response. What you are suggesting with pinging the container from the appropriate namespace may be worth a try but then there may be containers that do not allow ingress traffic while they are up and happy. So short of what Salvatore suggested in his earlier email (and I am not sure if that can be done without additions to Neutron), we are left with option 1. Keep in mind that users can choose not to enable the blocking option and things will be as they are right now. Would that be reasonable? Best, Mohammad Neil Jerram ---06/10/2016 09:25:59 AM---Hi Mohammad, Why is the blocking needed? Is it to report some kind of status back to From: Neil Jerram To: "OpenStack Development Mailing List (not for usage questions)" Date: 06/10/2016 09:25 AM Subject: Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron Port is Active Hi Mohammad, Why is the blocking needed? Is it to report some kind of status back to Docker/Kubernetes, or to allow some follow-on action to happen? When using networking-calico as the driver, I think that only option (1) would work, out of the options you've suggested below. (3) doesn't work, as you say, because Calico doesn't involve an L2 agent. Also Calico doesn't use the RPC message queue for reporting port status, because we've found that that message queue is in itself a scalability bottleneck. I guess another option would be for the using system to determine for itself when the port appears to be working, e.g. by the host pinging the container/pod's IP address. Regards, Neil On Wed, Jun 8, 2016 at 4:23 PM Mohammad Banikazemi wrote: For the Kuryr project, in order to support blocking until vifs are plugged in (that is adding config options similar to the following options define in Nova: vif_plugging_is_fatal and vif_plugging_timeout), we need to detect that the Neutron plugin being used is done with plugging a given vif. Here are a few options: 1- The simplest approach seems to be polling for the status of the Neutron port to become Active. (This may lead to scalability issues but short of having a specific goal for scalability, it is not clear that will be the case.) 2- Alternatively, We could subscribe to the message queue and wait for such a port update event. 3- It was also suggested that we could use l2 agent extension to detect such an event but that seems to limit us to certain Neutron plugins and therefore not acceptable. I was wondering if there are other and better options. Best, Mohammad __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
Re: [openstack-dev] [neutron] [designate] multi-tenancy in Neutron's DNS integration
"Hayes, Graham" <graham.ha...@hpe.com> wrote on 05/09/2016 04:08:07 PM: > ... > On 09/05/2016 20:55, Mike Spreitzer wrote: > ... > > Oh, right, the network gets to specify the rest of the FQDN. In my case > > I am interested in Neutron Ports on tenant networks. So with a per-port > > "hostname" (first label) and per-network "domain" (rest of the labels), > > I would get separation between tenants --- at least in the sense that > > there is no overlap in FQDNs. Will this work for private tenant networks? > > Yes, you could publish the records to Designate for this, or using the > internal dns resolution side of the integration. > > Pushing the records to designate would make them viewable globally > (anywhere the DNS servers are accessible) > > > > The other part of separation is that I do not want one tenant to even be > > able to look up FQDNs that belong to another tenant. Is this > > prohibition possible today? If not, is anyone else interested in it? > > Do you want to limit this to inside the tenant private network? if so, > just allowing users to set the dns_domain on a network, and not enabling > the external DNS plugin will work fine. Ah, that may be what I want. BTW, I am not planning to use Nova. I am planning to use Swarm and Kubernetes to create containers attached to Neutron private tenant networks. What DNS server would I configure those containers to use? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [designate] multi-tenancy in Neutron's DNS integration
"Hayes, Graham" <graham.ha...@hpe.com> wrote on 05/09/2016 03:00:34 PM: > From: "Hayes, Graham" <graham.ha...@hpe.com> > To: "OpenStack Development Mailing List (not for usage questions)" > <openstack-dev@lists.openstack.org> > Date: 05/09/2016 03:05 PM > Subject: Re: [openstack-dev] [neutron] [designate] multi-tenancy in > Neutron's DNS integration > > On 09/05/2016 19:21, Mike Spreitzer wrote: > > I just read > > http://docs.openstack.org/mitaka/networking-guide/adv-config-dns.htmland > , unless > > I missed something, it seems to be describing something that is not > > multi-tenant. I am focused on FQDNs for Neutron Ports. For those, only > > the "hostname" part (the first label, in official DNS jargon) is > > controllable by the Neutron user, the rest of the FQDN is fixed in > > Neutron configuration. Have I got that right? If so then I am > > surprised. I would have expected something that isolates tenants > > (projects) from one another. Is there any interest in such a thing? > > > > Thanks, > > Mike > > ... > > If you have per-project networks the integration can be done on a > project by project basis, with floating IPs assigned the name from > the port and the zone from the private network. Oh, right, the network gets to specify the rest of the FQDN. In my case I am interested in Neutron Ports on tenant networks. So with a per-port "hostname" (first label) and per-network "domain" (rest of the labels), I would get separation between tenants --- at least in the sense that there is no overlap in FQDNs. Will this work for private tenant networks? The other part of separation is that I do not want one tenant to even be able to look up FQDNs that belong to another tenant. Is this prohibition possible today? If not, is anyone else interested in it? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] [designate] multi-tenancy in Neutron's DNS integration
I just read http://docs.openstack.org/mitaka/networking-guide/adv-config-dns.html and, unless I missed something, it seems to be describing something that is not multi-tenant. I am focused on FQDNs for Neutron Ports. For those, only the "hostname" part (the first label, in official DNS jargon) is controllable by the Neutron user, the rest of the FQDN is fixed in Neutron configuration. Have I got that right? If so then I am surprised. I would have expected something that isolates tenants (projects) from one another. Is there any interest in such a thing? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [kuryr] Question for Antoni Puimedon about load balancers and overlay networks
I am looking at https://www.openstack.org/videos/video/project-kuryr-docker-delivered-kubernetes-next , around 28:00. You have said that overlay networks are involved, and are now talking about load balancers. Is this Neutron LBaaS? As far as I know, a Neutron LBaaS instance is "one-armed" --- both the VIP and the back endpoints have to be on the same Neutron network. But you seem to have all the k8s services on a single subnet. So I am having some trouble following exactly what is going on. Can you please elaborate? BTW, there is also some discussion of k8s multi-tenancy in the Kubernetes Networking SIG and the Kubernetes OpenStack SIG. Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack][neutron] Eliminating the DevStack layer
I like this, for a reason not mentioned. I am not a Neutron dev or operator and have never learned how to deploy Neutron; I have always driven it through DevStack. The documentation for that has never been adequate, and I have concluded it will never be adequate. With inadequate documentation, that layer isn't really doing its job anyway. If there is no devstack layer getting in my way, I am willing to learn how to deploy Neutron from what I suppose is better documentation than I have been reading. I understand that direct Neutron configuration is more troublesome than it should be. Eliminating the devstack Neutron layer will only increase the pressure to both improve the documentation of Neutron configuration and simplify the subject, both of which are Good Things (TM). Regards, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [kuryr] Did kuryr need to know about Docker's cluster store?
On Feb 5 I was given a tarchive of kuryr with an install script that configures the docker daemon to use consul as its cluster store. If I modify the config of docker to use etcd instead then do I need to change anything in Kuryr? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] - Changing the Neutron default security group rules
Kevin Bentonwrote on 03/02/2016 01:27:27 PM: > Does it at least also include the UUID, or is there no way to tell > from 'nova show'? No direct way to tell, as far as I can see. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] - Changing the Neutron default security group rules
"Sean M. Collins"wrote on 03/02/2016 01:16:52 PM: > Meaning your users are creating new security groups and naming them > "default" - so you have the "default" default (heh) and then the one > that they created named default? > > Are security group names in Nova-Net unqiue? I seem to recall that being > a difference between Nova-Net and Neutron, where security group names > are not unique in Neutron - hence the problem above. I have seen this happen in a variety of use cases. It does not bother me that security group names are scoped to tenants; other kinds of names also lack various kinds of uniqueness. I am really just raising a tiny, peripheral rant here. When I go in as admin to look at a problem, `nova show` identifies a Compute Instance's security group in a useless way. If only I could make `nova show` tell me the UUID instead of, or in addition to, the security group's name then I would be happy. Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] - Changing the Neutron default security group rules
"Sean M. Collins"wrote on 03/02/2016 12:38:29 PM: > I think that the default security group should be left as is - and users > should be trained that they should bring/create security groups with the > appropriate rules for their need. Could we at least make it less difficult to figure out which security group is attached to a Nova instance? Right now `nova show` says only that the security group is named "default" and guess what --- they are *all* named default! An admin looking at this is lost. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [devstack] [neutron] How to ask for linuxbridge instead of openvswitch
How would I modify the recipe at http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface to get linuxbridge instead of openvswitch? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] MTU configuration pain
BTW, regarding devstack: See https://bugs.launchpad.net/devstack/+bug/1532924. I have been trying to get the current code to work, following the ideas in https://specs.openstack.org/openstack/fuel-specs/specs/7.0/jumbo-frames-between-instances.html#proposed-change . It fails only at the last step: the MTU on the network interface inside the VM is still 1500. Regards, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] MTU configuration pain
Adam Lawsonwrote on 01/23/2016 02:27:46 PM: > For the sake of over-simplification, is there ever a reason to NOT > enable jumbo frames in a cloud/SDN context where most of the traffic > is between virtual elements that all support it? I understand that > some switches do not support it and traffic form the web doesn't > support it either but besides that, seems like a default > "jumboframes = 1" concept would work just fine to me. > > Then again I'm all about making OpenStack easier to consume so my > ideas tend to gloss over special use cases with special requirements. Regardless of the default, there needs to be clear documentation on what to do for those of us who can not use jumbo frames, and it needs to work. That goes for production deployers and also for developers using DevStack. Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [devstack] [neutron] Progress, and current problems
I have been trying to get http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface working on Ubuntu 14.04 (updated to the latest, which includes kernel 3.19). With the latest sources and a sufficiently recent OVS, the one-node install pretty much works! I added to local.conf only: (a) IP_VERSION=4, to avoid issues with v6, and (b) some settings to get logs saved to files without coloring. In particular, what I have done regarding OVS is to download 2.4 and build the .deb packages according to the instructions. I used the kernel datapath from the linux kernel 3.19. I installed the openvswitch-common and openvswitch-switch .debs that I built. I think the doc should be enhanced to discuss IPv6. I tried once without the `IP_VERSION=4`, and the resulting IPv6 config was not good. IMHO the default should be to keeps logs in files without coloring, because that is the most user-friendly. I think that for DevStack the defaults should be friendly to newbies, since this is where they start. I then went on to http://docs.openstack.org/developer/devstack/guides/neutron.html#adding-additional-compute-nodes , and ran into MTU problems. All MTUs come out to be 1500, which just does not work (due to the tunneling overhead). I then tried a fresh DevStack install with a revised local.conf derived from https://specs.openstack.org/openstack/fuel-specs/specs/7.0/jumbo-frames-between-instances.html#proposed-change . Specifically, this is what I added to my local.conf for MTU issues: [[post-config|$NEUTRON_CONF]] [DEFAULT] advertise_mtu=True network_device_mtu=1450 [[post-config|/$Q_PLUGIN_CONF_FILE]] [ml2] physical_network_mtus = public:1450 That got me MTUs of 1450 on the router and DHCP ports but not the VM plumbing. Perhaps I read too much into the word "keep". Then I added network_device_mtu = 65000 to the DEFAULT section of nova.conf, then restarted all the nova processes. I created a new VM after that. The result was that the qbr..., qvo..., qvb..., and tap... network interfaces in the main netns have MTU of 65000 --- and eth0 inside the VM has an MTU of 1500. So then I changed the network_device_mtu setting in nova.conf to 1450, restarted all the nova processes again, and made another new VM. Now the qbr..., qvo..., qvb..., and tap... network interfaces in the main netns have MTU of 1450 but the VM still got an MTU of 1500 on its eth0. In my logs directory, `grep 'Running command' *.log | grep set | grep -i mtu` finds only the setting of the MTUs of the router and DHCP stuff --- nothing for any of the VMs. Regards, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] [neutron] Procedure for back-porting a bug fix to stable/liberty branch of a Neutron script in DevStack?
> From: Kyle Mestery <mest...@mestery.com> > To: "OpenStack Development Mailing List (not for usage questions)" > <openstack-dev@lists.openstack.org> > Date: 01/03/2016 02:15 PM > Subject: Re: [openstack-dev] [devstack] [neutron] Procedure for > back-porting a bug fix to stable/liberty branch of a Neutron script > in DevStack? > > On Sun, Jan 3, 2016 at 4:01 PM, Mike Spreitzer <mspre...@us.ibm.com> wrote: > I would like to back-port https://review.openstack.org/#/c/242721/ > --- which fixed https://bugs.launchpad.net/devstack/+bug/1469596--- > to stable/liberty. I have contributed to main line development > before, but not stable branches. I see that http:// > docs.openstack.org/infra/manual/developers.htmldoes not specifically > address this case. What is the procedure to follow? > The best place to look is in the project team guide [1], > specifically the section on proposing fixes. > > [1] http://docs.openstack.org/project-team-guide/stable-branches.html > [2] http://docs.openstack.org/project-team-guide/stable-branches.html#proposing-fixes Reference [2] says I should $ git checkout stable/tango but that doesn't work. Here is a typescript of how it goes wrong: mjs10:devstack mspreitz$ git status On branch master Your branch is up-to-date with 'origin/master'. nothing to commit, working directory clean mjs10:devstack mspreitz$ git pull Already up-to-date. mjs10:devstack mspreitz$ git remote -v gerrit ssh://mspre...@review.openstack.org:29418/openstack-dev/devstack.git (fetch) gerrit ssh://mspre...@review.openstack.org:29418/openstack-dev/devstack.git (push) origin https://github.com/openstack-dev/devstack.git (fetch) origin https://github.com/openstack-dev/devstack.git (push) mjs10:devstack mspreitz$ git checkout stable/liberty error: pathspec 'stable/liberty' did not match any file(s) known to git. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] [neutron] Procedure for back-porting a bug fix to stable/liberty branch of a Neutron script in DevStack?
> From: Kyle Mestery <mest...@mestery.com> > To: "OpenStack Development Mailing List (not for usage questions)" > <openstack-dev@lists.openstack.org> > Date: 01/03/2016 02:15 PM > Subject: Re: [openstack-dev] [devstack] [neutron] Procedure for > back-porting a bug fix to stable/liberty branch of a Neutron script > in DevStack? > > On Sun, Jan 3, 2016 at 4:01 PM, Mike Spreitzer <mspre...@us.ibm.com> wrote: > I would like to back-port https://review.openstack.org/#/c/242721/ > --- which fixed https://bugs.launchpad.net/devstack/+bug/1469596--- > to stable/liberty. I have contributed to main line development > before, but not stable branches. I see that http:// > docs.openstack.org/infra/manual/developers.htmldoes not specifically > address this case. What is the procedure to follow? > The best place to look is in the project team guide [1], > specifically the section on proposing fixes. > > [1] http://docs.openstack.org/project-team-guide/stable-branches.html > [2] http://docs.openstack.org/project-team-guide/stable- > branches.html#proposing-fixes Regarding the mechanics, I was given the following alternate recipe in a discussion on #openstack-neutron; I assume it is pretty much equivalent. "do something like 'git checkout -b libery/backport/### remotes/origin/stable/liberty' then 'git pull' then 'git review -X ###' then push it for review" Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [devstack] [neutron] Procedure for back-porting a bug fix to stable/liberty branch of a Neutron script in DevStack?
I would like to back-port https://review.openstack.org/#/c/242721/ --- which fixed https://bugs.launchpad.net/devstack/+bug/1469596 --- to stable/liberty. I have contributed to main line development before, but not stable branches. I see that http://docs.openstack.org/infra/manual/developers.html does not specifically address this case. What is the procedure to follow? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Multiple locations for documentation of features
> From: Henry Gessau> To: "OpenStack Development Mailing List (not for usage questions)" > > Date: 12/04/2015 02:23 PM > Subject: Re: [openstack-dev] [neutron] Multiple locations for > documentation of features > > Sean M. Collins wrote: > > I've noticed that a lot of features are now being documented as RSTs > > inside of devref. Like the following: > > > > https://review.openstack.org/#/c/251859/ > > > > But there are lots already present. Can someone point out to me what the > > criteria is for these documents? I am a little confused about the > > relationship between neutron-specs, RFE bugs, and some features being > > documented in devref. Especially when a review includes the actual code, > > then a new RST file in devref - wasn't that what specs were for? > > Here is how I would like to see things ending up: > > 1. RFE: "I want X" > 2. Spec: "I plan to implement X like this" > 3. devref: "How X is implemented and how to extend it" > 4. OS docs: "API and guide for using X" > > Once X is implemented I don't want to have to go to 1 or 2 to find information > on it. The devref may have a lot of content from the spec, but the spec is not > maintained and the implementation may differ in some ways. The devref should > be kept current with refactorings, etc. of the implementation. Lots cheers from me too. Let me add one thing: "the spec is not maintained" is a remaining process bug. A spec by itself is a very useful thing. It is the first thing to read when trying to understand the implementation. How about we resolve that devref includes a maintained version of the spec? Regards, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] The OVS and Provider Networks section of the guide to Neutron in DevStack
First let me make sure I understand what this section ( http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks ) is trying to say. The second paragraph is saying that some infrastructure provider has allocated a VLAN tag (and the later display of localrc contents uses 2010 as that tag) and an address range (later seen to be 203.0.113.0/24) to use on that VLAN on provider_net. The exhibited localrc contents also say that tenant networks can be created, using VLAN tags drawn from the range 3001:4000. Those localrc contents do not explicitly say which ethernet --- provider_net or control_plane --- will carry those tenant VLANs. Which ethernet do you think those localrc contents imply will carry the tenant VLANs? Second, I tested this using stable/liberty on Ubuntu 14.04.03, and got a few surprises. I did exactly as described by that section, starting from a "physical network setup" as described there (but it was actually virtual, using VirtualBox), and using the localrc contents exhibited there except patched to fix bugs https://bugs.launchpad.net/devstack/+bug/1508195 and https://bugs.launchpad.net/devstack/+bug/1508202 (the latter by sadly stipulating IP_VERSION=4). After creating the controller, compute1, and compute2 I looked at `ovs-vsctl show` on the controller, and found that br-tun had two VxLan ports (see display below). This surprised me because I thought that VxLan can itself handle multiple hosts, it is not just point-to-point. Bridge br-tun fail_mode: secure Port "vxlan-0a03" Interface "vxlan-0a03" type: vxlan options: {df_default="true", in_key=flow, local_ip="10.0.0.2", out_key=flow, remote_ip="10.0.0.3"} Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port "vxlan-0a04" Interface "vxlan-0a04" type: vxlan options: {df_default="true", in_key=flow, local_ip="10.0.0.2", out_key=flow, remote_ip="10.0.0.4"} Port br-tun Interface br-tun type: internal My second surprise is that the path from controller to the VM's host goes over the control_plane network --- somewhat contrary to its name! My third surprise came after I made a VM on the provider network and tried to ping it from the router. I got no response. A little investigation shows that the router's ARP request for the VM gets lost somewhere between br-int and br-tun. `tcpdump -nne -i br-int` on the controller shows the ARP requests appearing with VLAN tag 1 (which is right, the external VLAN tags are translated to internal host-local tags on br-int), and no ARP replies. OTOH, `tcpdump -nne -i br-tun` does not even find the ARP requests. Digging a little deeper, I looked at the OpenFlow tables on the controller. It looks like the flow table for br-tun does indeed prescribe that all traffic coming from outside be dropped! Here is the output from `ovs-appctl bridge/dump-flows br-tun`: duration=23916s, priority=1, n_packets=0, n_bytes=0, priority=1,in_port=3,actions=resubmit(,4) duration=39904s, priority=1, n_packets=82, n_bytes=7025, priority=1,in_port=1,actions=resubmit(,2) duration=36711s, priority=1, n_packets=0, n_bytes=0, priority=1,in_port=2,actions=resubmit(,4) duration=39904s, priority=0, n_packets=7, n_bytes=558, priority=0,actions=drop table_id=2, duration=39904s, priority=0, n_packets=0, n_bytes=0, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00,actions=resubmit(,20) table_id=2, duration=39904s, priority=0, n_packets=82, n_bytes=7025, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00,actions=resubmit(,22) table_id=3, duration=39904s, priority=0, n_packets=0, n_bytes=0, priority=0,actions=drop table_id=4, duration=39904s, priority=0, n_packets=0, n_bytes=0, priority=0,actions=drop table_id=6, duration=39904s, priority=0, n_packets=0, n_bytes=0, priority=0,actions=drop table_id=10, duration=39904s, priority=1, n_packets=0, n_bytes=0, priority=1,actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xb64bc7164f7e1137,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1 table_id=20, duration=39904s, priority=0, n_packets=0, n_bytes=0, priority=0,actions=resubmit(,22) table_id=22, duration=39904s, priority=0, n_packets=82, n_bytes=7025, priority=0,actions=drop table_id=254, duration=39904s, priority=0, n_packets=0, n_bytes=0, priority=0,reg0=0x3,actions=drop table_id=254, duration=39904s, priority=0, n_packets=1, n_bytes=90, priority=0,reg0=0x1,actions=controller(reason=no_match) table_id=254, duration=39904s, priority=0, n_packets=0, n_bytes=0, priority=0,reg0=0x2,actions=drop By examining `ovs-vsctl list Interface` I can see that patch-int is OpenFlow
Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron
"Sean M. Collins" <s...@coreitpro.com> wrote on 10/16/2015 01:03:53 PM: > From: "Sean M. Collins" <s...@coreitpro.com> > To: "OpenStack Development Mailing List (not for usage questions)" > <openstack-dev@lists.openstack.org> > Date: 10/16/2015 01:12 PM > Subject: Re: [openstack-dev] [devstack] [neutron] A larger batch of > questions about configuring DevStack to use Neutron > > On Fri, Oct 16, 2015 at 02:02:57AM EDT, Mike Spreitzer wrote: > > Now I have tested the first section (taking the latest of both changes) > > using stable/liberty, and it failed because br-ex was not created > > automatically. I opened https://bugs.launchpad.net/devstack/+bug/1506733 > > > > I think Kilo did not have this problem. > > > > Ok. I have assigned myself the bug. I will be sightseeing with my partner > Caroline in Tokyo next week before the summit, so it may take me a > little bit of time to get around to it, but I will try and follow up on > the bug after the summit. Thanks. I am going to do some more tests in the interim, to see what I can add. Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron
> From: Mike Spreitzer/Watson/IBM@IBMUS > To: "OpenStack Development Mailing List \(not for usage questions\)" > <openstack-dev@lists.openstack.org> > Date: 10/13/2015 03:39 AM > Subject: Re: [openstack-dev] [devstack] [neutron] A larger batch of > questions about configuring DevStack to use Neutron > > > From: "Sean M. Collins" <s...@coreitpro.com> > > To: "OpenStack Development Mailing List (not for usage questions)" > > <openstack-dev@lists.openstack.org> > > Date: 10/12/2015 11:34 AM > > Subject: Re: [openstack-dev] [devstack] [neutron] A larger batch of > > questions about configuring DevStack to use Neutron > > > > On Thu, Oct 08, 2015 at 01:47:31PM EDT, Mike Spreitzer wrote: > > > .. > > > > > In the section > > > > > http://docs.openstack.org/developer/devstack/guides/ > > > > neutron.html#using-neutron-with-a-single-interface > > > > > there is a helpful display of localrc contents. It says, among other > > > > > things, > > > > > > > > > >OVS_PHYSICAL_BRIDGE=br-ex > > > > >PUBLIC_BRIDGE=br-ex > > > > > > > > > > In the next top-level section, > > > > > http://docs.openstack.org/developer/devstack/guides/ > > > > neutron.html#using-neutron-with-multiple-interfaces > > > > > , there is no display of revised localrc contents and no mention of > > > > > changing either bridge setting. That is an oversight, right? > > > > > > > > No, this is deliberate. Each section is meant to be independent, since > > > > each networking configuration and corresponding DevStack configuration > > > > is different. Of course, this may need to be explicitly stated in the > > > > guide, so there is always room for improvement. > > > > > > I am not quite sure I understand your answer. Is the intent that I can > > > read only one section, ignore all the others, and that will tellme how to > > > use DevStack to produce that section's configuration? If so > then it would > > > be good if each section had a display of all the necessary localrc > > > contents. > > > > Agreed. This is a failure on my part, because I was pasting in only > > parts of my localrc (since it came out of a live lab environment). I've > > started pushing patches to correct this. > > > > > > > I have started over, from exactly the picture drawn at the start of the > > > doc. That has produced a configuration that mostly works. However, I > > > tried creating a VM connected to the public network, and that failed for > > > lack of a Neutron DHCP server there. > > > > The public network is used for floating IPs. The L3 agent creates NAT > > rules to intercept traffic on the public network and NAT it back to a > > guest instance that has the floating IP allocated to it. > > > > The behavior for when a guest is directly attached to the public > > network, I sort of forget what happens exactly but I do know that it > > doesn't work from experience - most likely because the network is not > > configured as a flat network. It will not receive a DHCP lease from the > > external router. > > > > > I am going to work out how to change > > > that, and am willing to contribute an update to this doc. Wouldyou want > > > that in this section --- in which case this section needs to specify that > > > the provider DOES NOT already have DHCP service on the hardware network > > > --- or as a new section? > > > > No, I think we should maybe have a warning or something that the > > external network will be used for Floating IPs, and that guest instances > > should not be directly attached to that network. > > > > > > > > > > > > Looking at > > > > > http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html(or > > > , in > > > > > former days, the doc now preserved at > > > > > http://docs.ocselected.org/openstack-manuals/kilo/networking- > > > > guide/content/under_the_hood_openvswitch.html > > > > > ) I see the name br-ex used for $PUBLIC_BRIDGE--- not > > > $OVS_PHYSICAL_BRIDGE > > > > > , right? Wouldn't it be less confusing if > > > > > http://docs.openstack.org/developer/devstack/guides/neutron.htmluseda > > > > > > > > name other than "br-ex"
Re: [openstack-dev] [devstack] A few questions on configuring DevStack for Neutron
Matt Riedemannwrote on 10/08/2015 09:48:33 PM: > On 10/8/2015 11:38 AM, Sean M. Collins wrote: > > Please see my response here: > > > > http://lists.openstack.org/pipermail/openstack-dev/2015-October/076251.html > > > > In the future, do not create multiple threads since responses will get > > lost > > > > Maybe a dumb question, but couldn't people copy the localrc from the > gate-tempest-dsvm-neutron-full job, i.e.: > > http://logs.openstack.org/20/231720/1/check/gate-tempest-dsvm- > neutron-full/f418dc8/logs/localrc.txt.gz Matt, thanks for the reminder. I was pointed at one of those once before, but do not remember how to find them in general. To be useful in http://docs.openstack.org/developer/devstack/guides/neutron.html we need to identify which section they illuminate, or add another section with appropriate explanation. Which would it be? Sean, you said that those URLs are tribal knowledge. Would you recommend documenting them and, if so, where? I see that the one Matt cited is part of a comprehensive archive from a tempest run, and there is even some explanatory material included within that run's archive. Would it be possible and appropriate for the DevStack Neutron guide to point to some documentation that describes these archives? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron
> From: "Sean M. Collins" <s...@coreitpro.com> > To: "OpenStack Development Mailing List (not for usage questions)" > <openstack-dev@lists.openstack.org> > Date: 10/12/2015 11:34 AM > Subject: Re: [openstack-dev] [devstack] [neutron] A larger batch of > questions about configuring DevStack to use Neutron > > On Thu, Oct 08, 2015 at 01:47:31PM EDT, Mike Spreitzer wrote: > > .. > > > > In the section > > > > http://docs.openstack.org/developer/devstack/guides/ > > > neutron.html#using-neutron-with-a-single-interface > > > > there is a helpful display of localrc contents. It says, among other > > > > things, > > > > > > > >OVS_PHYSICAL_BRIDGE=br-ex > > > >PUBLIC_BRIDGE=br-ex > > > > > > > > In the next top-level section, > > > > http://docs.openstack.org/developer/devstack/guides/ > > > neutron.html#using-neutron-with-multiple-interfaces > > > > , there is no display of revised localrc contents and no mention of > > > > changing either bridge setting. That is an oversight, right? > > > > > > No, this is deliberate. Each section is meant to be independent, since > > > each networking configuration and corresponding DevStack configuration > > > is different. Of course, this may need to be explicitly stated in the > > > guide, so there is always room for improvement. > > > > I am not quite sure I understand your answer. Is the intent that I can > > read only one section, ignore all the others, and that will tell me how to > > use DevStack to produce that section's configuration? If so then it would > > be good if each section had a display of all the necessary localrc > > contents. > > Agreed. This is a failure on my part, because I was pasting in only > parts of my localrc (since it came out of a live lab environment). I've > started pushing patches to correct this. > > > > I have started over, from exactly the picture drawn at the start of the > > doc. That has produced a configuration that mostly works. However, I > > tried creating a VM connected to the public network, and that failed for > > lack of a Neutron DHCP server there. > > The public network is used for floating IPs. The L3 agent creates NAT > rules to intercept traffic on the public network and NAT it back to a > guest instance that has the floating IP allocated to it. > > The behavior for when a guest is directly attached to the public > network, I sort of forget what happens exactly but I do know that it > doesn't work from experience - most likely because the network is not > configured as a flat network. It will not receive a DHCP lease from the > external router. > > > I am going to work out how to change > > that, and am willing to contribute an update to this doc. Would you want > > that in this section --- in which case this section needs to specify that > > the provider DOES NOT already have DHCP service on the hardware network > > --- or as a new section? > > No, I think we should maybe have a warning or something that the > external network will be used for Floating IPs, and that guest instances > should not be directly attached to that network. > > > > > > > > > Looking at > > > > http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html(or > > , in > > > > former days, the doc now preserved at > > > > http://docs.ocselected.org/openstack-manuals/kilo/networking- > > > guide/content/under_the_hood_openvswitch.html > > > > ) I see the name br-ex used for $PUBLIC_BRIDGE--- not > > $OVS_PHYSICAL_BRIDGE > > > > , right? Wouldn't it be less confusing if > > > > http://docs.openstack.org/developer/devstack/guides/neutron.htmlused a > > > > > > name other than "br-ex" for the exhibited commands that apply to > > > > $OVS_PHYSICAL_BRIDGE? > > > > > > No, this is deliberate - br-ex is the bridge that is used for external > > > network traffic - such as floating IPs and public IP address ranges. On > > > the network node, a physical interface is attached to br-ex so that > > > traffic will flow. > > > > > > PUBLIC_BRIDGE is a carryover from DevStack's Nova-Network support and is > > > used in some places, with OVS_PHYSICAL_BRIDGE being used by DevStack's > > > Neutron support, for the Open vSwitch driver specifically. They are two > > > variables that for the most part serve the same pu
Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron
Thanks, i will review soon. BTW, i am interested in creating a config in which a Compute Instance can be created on an external network. Thanks, Mike Sean M. Collins --- Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron --- From:"Sean M. Collins" <s...@coreitpro.com>To:"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org>Date:Mon, Oct 12, 2015 11:34Subject:Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron On Thu, Oct 08, 2015 at 01:47:31PM EDT, Mike Spreitzer wrote: > .. > > > In the section > > > http://docs.openstack.org/developer/devstack/guides/ > > neutron.html#using-neutron-with-a-single-interface > > > there is a helpful display of localrc contents. It says, among other > > > things, > > > > > > OVS_PHYSICAL_BRIDGE=br-ex > > > PUBLIC_BRIDGE=br-ex > > > > > > In the next top-level section, > > > http://docs.openstack.org/developer/devstack/guides/ > > neutron.html#using-neutron-with-multiple-interfaces > > > , there is no display of revised localrc contents and no mention of > > > changing either bridge setting. That is an oversight, right? > > > > No, this is deliberate. Each section is meant to be independent, since > > each networking configuration and corresponding DevStack configuration > > is different. Of course, this may need to be explicitly stated in the > > guide, so there is always room for improvement. > > I am not quite sure I understand your answer. Is the intent that I can > read only one section, ignore all the others, and that will tell me how to > use DevStack to produce that section's configuration? If so then it would > be good if each section had a display of all the necessary localrc > contents. Agreed. This is a failure on my part, because I was pasting in only parts of my localrc (since it came out of a live lab environment). I've started pushing patches to correct this. > I have started over, from exactly the picture drawn at the start of the > doc. That has produced a configuration that mostly works. However, I > tried creating a VM connected to the public network, and that failed for > lack of a Neutron DHCP server there. The public network is used for floating IPs. The L3 agent creates NAT rules to intercept traffic on the public network and NAT it back to a guest instance that has the floating IP allocated to it. The behavior for when a guest is directly attached to the public network, I sort of forget what happens exactly but I do know that it doesn't work from experience - most likely because the network is not configured as a flat network. It will not receive a DHCP lease from the external router. > I am going to work out how to change > that, and am willing to contribute an update to this doc. Would you want > that in this section --- in which case this section needs to specify that > the provider DOES NOT already have DHCP service on the hardware network > --- or as a new section? No, I think we should maybe have a warning or something that the external network will be used for Floating IPs, and that guest instances should not be directly attached to that network. > > > > > > Looking at > > > http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html(or > , in > > > former days, the doc now preserved at > > > http://docs.ocselected.org/openstack-manuals/kilo/networking- > > guide/content/under_the_hood_openvswitch.html > > > ) I see the name br-ex used for $PUBLIC_BRIDGE--- not > $OVS_PHYSICAL_BRIDGE > > > , right? Wouldn't it be less confusing if > > > http://docs.openstack.org/developer/devstack/guides/neutron.htmlused a > > > > name other than "br-ex" for the exhibited commands that apply to > > > $OVS_PHYSICAL_BRIDGE? > > > > No, this is deliberate - br-ex is the bridge that is used for external > > network traffic - such as floating IPs and public IP address ranges. On > > the network node, a physical interface is attached to br-ex so that > > traffic will flow. > > > > PUBLIC_BRIDGE is a carryover from DevStack's Nova-Network support and is > > used in some places, with OVS_PHYSICAL_BRIDGE being used by DevStack's > > Neutron support, for the Open vSwitch driver specifically. They are two > > variables that for the most part serve the same purpose. Frankly, > > DevStack has a lot of problems with configuration knobs, and > > PUBLIC_BRIDGE and OVS_PHYSICAL_BRIDGE is just a symptom. > >
Re: [openstack-dev] [all] Recording little everyday OpenStack successes
Thierry Carrezwrote on 10/09/2015 05:42:49 AM: ... > So whenever you feel like you made progress, or had a little success in > your OpenStack adventures, or have some joyful moment to share, just > throw the following message on your local IRC channel: > > #success [Your message here] > > The openstackstatus bot will take that and record it on this wiki page: > > https://wiki.openstack.org/wiki/Successes > > We'll add a few of those every week to the weekly newsletter (as part of > the developer digest that we reecently added there). > > Caveats: Obviously that only works on channels where openstackstatus is > present (the official OpenStack IRC channels), and we may remove entries > that are off-topic or spam. > > So... please use #success liberally and record lttle everyday OpenStack > successes. Share the joy and make the OpenStack community a happy place. Great. I am about to contribute one myself. Lucky I noticed this email. How will the word get out to those who did not? How about a pointer to instructions on the Successes page? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron
> From: "Sean M. Collins" <s...@coreitpro.com> ... > > On Tue, Oct 06, 2015 at 11:25:03AM EDT, Mike Spreitzer wrote: > > [Sorry, but I do not know if the thundering silence is because these > > questions are too hard, too easy, grossly off-topic, or simply because > > nobody cares.] > > You sent your first e-mail on a Saturday. I saw it and flagged it for > reply, but have not had a chance yet. It's only Tuesday. I do care and > your questions are important. I will say though that it's a little > difficult to answer your e-mail because of formatting and your thoughts > seem to jump around. This is not intended as a personal criticism, it's > just a little difficult to follow your e-mail in order to reply. Thanks, I am glad somebody cares. I used different subject lines because I was suspecting that I did not flag them correctly. I see now that I was just too impatient. .. > > In the section > > http://docs.openstack.org/developer/devstack/guides/ > neutron.html#using-neutron-with-a-single-interface > > there is a helpful display of localrc contents. It says, among other > > things, > > > >OVS_PHYSICAL_BRIDGE=br-ex > >PUBLIC_BRIDGE=br-ex > > > > In the next top-level section, > > http://docs.openstack.org/developer/devstack/guides/ > neutron.html#using-neutron-with-multiple-interfaces > > , there is no display of revised localrc contents and no mention of > > changing either bridge setting. That is an oversight, right? > > No, this is deliberate. Each section is meant to be independent, since > each networking configuration and corresponding DevStack configuration > is different. Of course, this may need to be explicitly stated in the > guide, so there is always room for improvement. I am not quite sure I understand your answer. Is the intent that I can read only one section, ignore all the others, and that will tell me how to use DevStack to produce that section's configuration? If so then it would be good if each section had a display of all the necessary localrc contents. OTOH, if some sections build on other sections, it would be good if the dependent sections display the localrc contents that differ. Right now, the section at http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces does not display any localrc contents at all. > For example, There needs > to be some editing done for that doc - the part about disabling the > firewall is just dropped in the middle of the doc and breaks the flow - > among other things. This is obviously not helpful to a new reader and we > need to fix. > > > > I am > > guessing I need to set OVS_PHYSICAL_BRIDGEand PUBLIC_BRIDGEto different > > values, and the exhibited `ovs-vsctl` commands in this section apply to > > $OVS_PHYSICAL_BRIDGE. Is that right? Are there other revisions I need to > > make to localrc? > > No, this is not correct. > > What does your networking layout look like on the DevStack node that you > are trying to configure? I have started over, from exactly the picture drawn at the start of the doc. That has produced a configuration that mostly works. However, I tried creating a VM connected to the public network, and that failed for lack of a Neutron DHCP server there. I am going to work out how to change that, and am willing to contribute an update to this doc. Would you want that in this section --- in which case this section needs to specify that the provider DOES NOT already have DHCP service on the hardware network --- or as a new section? > > > > > > Looking at > > http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html(or , in > > former days, the doc now preserved at > > http://docs.ocselected.org/openstack-manuals/kilo/networking- > guide/content/under_the_hood_openvswitch.html > > ) I see the name br-ex used for $PUBLIC_BRIDGE--- not $OVS_PHYSICAL_BRIDGE > > , right? Wouldn't it be less confusing if > > http://docs.openstack.org/developer/devstack/guides/neutron.htmlused a > > name other than "br-ex" for the exhibited commands that apply to > > $OVS_PHYSICAL_BRIDGE? > > No, this is deliberate - br-ex is the bridge that is used for external > network traffic - such as floating IPs and public IP address ranges. On > the network node, a physical interface is attached to br-ex so that > traffic will flow. > > PUBLIC_BRIDGE is a carryover from DevStack's Nova-Network support and is > used in some places, with OVS_PHYSICAL_BRIDGE being used by DevStack's > Neutron support, for the Open vSwitch driver specifically. They are two > variables that for the most part serve the same p
[openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron
[Sorry, but I do not know if the thundering silence is because these questions are too hard, too easy, grossly off-topic, or simply because nobody cares.] I have been looking at http://docs.openstack.org/developer/devstack/guides/neutron.htmland wonder about a few things. In the section http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface there is a helpful display of localrc contents. It says, among other things, OVS_PHYSICAL_BRIDGE=br-ex PUBLIC_BRIDGE=br-ex In the next top-level section, http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces , there is no display of revised localrc contents and no mention of changing either bridge setting. That is an oversight, right? I am guessing I need to set OVS_PHYSICAL_BRIDGEand PUBLIC_BRIDGEto different values, and the exhibited `ovs-vsctl` commands in this section apply to $OVS_PHYSICAL_BRIDGE. Is that right? Are there other revisions I need to make to localrc? Looking at http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html(or, in former days, the doc now preserved at http://docs.ocselected.org/openstack-manuals/kilo/networking-guide/content/under_the_hood_openvswitch.html ) I see the name br-ex used for $PUBLIC_BRIDGE--- not $OVS_PHYSICAL_BRIDGE , right? Wouldn't it be less confusing if http://docs.openstack.org/developer/devstack/guides/neutron.htmlused a name other than "br-ex" for the exhibited commands that apply to $OVS_PHYSICAL_BRIDGE? The section http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch builds on http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces NOT http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface --- right? Could I stop after reading that section, or must I go on to http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks ? The exhibited localrc contents in section http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface include both of these: Q_L3_ENABLED=True Q_USE_PROVIDERNET_FOR_PUBLIC=True and nothing gainsays either of them until section http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks --- where we first see Q_L3_ENABLED=False Is it true that all the other sections want both Q_L3_ENABLEDand Q_USE_PROVIDERNET_FOR_PUBLICto be True? I tried adding IPv6 support to the recipe of the first section ( http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface ). I added this to my localrc: IP_VERSION=4+6 IPV6_PUBLIC_RANGE=fddf:2::/64 IPV6_PUBLIC_NETWORK_GATEWAY=fddf:2::1 IPV6_ROUTER_GW_IP=fddf:2::231 At first I had tried setting a different set of IPv6 variables (having only IP_VERSION in common with what I exhibit here), but found those: (a) duplicated the defaults and (b) caused problems due to lack of the ones I mention here. Even the ones mentioned here led to a problem. There is a bit of scripging that replaces my setting for IPV6_ROUTER_GW_IP with something dug out of Neutron. That went wrong. It replaced my setting with fddf:2::2, but that address was already in use by something else. Thanks, Mike Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [devstack] A few questions on configuring DevStack for Neutron
[Apologies for re-posting, but I botched the subject line the first time and know that people use filters.] I have been looking at http://docs.openstack.org/developer/devstack/guides/neutron.html and wonder about a few things. In the section http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface there is a helpful display of localrc contents. It says, among other things, OVS_PHYSICAL_BRIDGE=br-ex PUBLIC_BRIDGE=br-ex In the next top-level section, http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces , there is no display of revised localrc contents and no mention of changing either bridge setting. That is an oversight, right? I am guessing I need to set OVS_PHYSICAL_BRIDGEand PUBLIC_BRIDGEto different values, and the exhibited `ovs-vsctl` commands in this section apply to $OVS_PHYSICAL_BRIDGE. Is that right? Are there other revisions I need to make to localrc? Looking at http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html (or, in former days, the doc now preserved at http://docs.ocselected.org/openstack-manuals/kilo/networking-guide/content/under_the_hood_openvswitch.html ) I see the name br-ex used for $PUBLIC_BRIDGE--- not $OVS_PHYSICAL_BRIDGE , right? Wouldn't it be less confusing if http://docs.openstack.org/developer/devstack/guides/neutron.html used a name other than "br-ex" for the exhibited commands that apply to $OVS_PHYSICAL_BRIDGE? The section http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch builds on http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces NOT http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface --- right? Could I stop after reading that section, or must I go on to http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks ? The exhibited localrc contents in section http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface include both of these: Q_L3_ENABLED=True Q_USE_PROVIDERNET_FOR_PUBLIC=True and nothing gainsays either of them until section http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks --- where we first see Q_L3_ENABLED=False Is it true that all the other sections want both Q_L3_ENABLEDand Q_USE_PROVIDERNET_FOR_PUBLICto be True? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] A few questions on configuring DevStack for Neutron
I have been looking at http://docs.openstack.org/developer/devstack/guides/neutron.html and wonder about a few things. In the section http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface there is a helpful display of localrc contents. It says, among other things, OVS_PHYSICAL_BRIDGE=br-ex PUBLIC_BRIDGE=br-ex In the next top-level section, http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces , there is no display of revised localrc contents and no mention of changing either bridge setting. That is an oversight, right? I am guessing I need to set OVS_PHYSICAL_BRIDGE and PUBLIC_BRIDGE to different values, and the exhibited `ovs-vsctl` commands in this section apply to $OVS_PHYSICAL_BRIDGE. Is that right? Are there other revisions I need to make to localrc? Looking at http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html (or, in former days, the doc now preserved at http://docs.ocselected.org/openstack-manuals/kilo/networking-guide/content/under_the_hood_openvswitch.html ) I see the name br-ex used for $PUBLIC_BRIDGE --- not $ OVS_PHYSICAL_BRIDGE, right? Wouldn't it be less confusing if http://docs.openstack.org/developer/devstack/guides/neutron.html used a name other than "br-ex" for the exhibited commands that apply to $OVS_PHYSICAL_BRIDGE? The section http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch builds on http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces NOT http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface --- right? Could I stop after reading that section, or must I go on to http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks ? The exhibited localrc contents in section http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface include both of these: Q_L3_ENABLED=True Q_USE_PROVIDERNET_FOR_PUBLIC=True and nothing gainsays either of them until section http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks --- where we first see Q_L3_ENABLED=False Is it true that all the other sections want both Q_L3_ENABLED and Q_USE_PROVIDERNET_FOR_PUBLIC to be True ? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] -1 due to line length violation in commit messages
> From: Gorka Eguileor> To: "OpenStack Development Mailing List (not for usage questions)" > > Date: 09/29/2015 07:34 AM > Subject: Re: [openstack-dev] [all] -1 due to line length violation > in commit messages ... > Since we are not all native speakers expecting everyone to realize that > difference - which is completely right - may be a little optimistic, > moreover considering that parts of those guidelines may even be written > by non natives. > > Let's say I interpret all "should" instances in that guideline as rules > that don't need to be strictly enforced, I see that the Change-Id > "should not be changed when rebasing" - this one would certainly be fun > to watch if we didn't follow it - the blueprint "should give the name of > a Launchpad blueprint" - I don't know any core that would not -1 a patch > if he notices the BP reference missing - and machine targeted metadata > "should all be grouped together at the end of the commit message" - this > one everyone follows instinctively, so no problem. > > And if we look at the i18n guidelines, almost everything is using > should, but on reviews these are treated as strict *must* because of the > implications. > > Anyway, it's a matter of opinion and afaik in Cinder we don't even have > a real problem with downvoting for the commit message length, I don't > see more than 1 every couple of months or so. Other communities have solved this by explicit reference to a standard defining terms like "must" and "should". Regards, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [magnum]swarm + compose = k8s?
> From: 王华> To: "OpenStack Development Mailing List (not for usage questions)" > > Date: 09/28/2015 11:34 PM > Subject: [openstack-dev] [magnum]swarm + compose = k8s? > > Hi folks, > > Magnum now exposes service, pod, etc to users in kubernetes coe, but > exposes container in swarm coe. As I know, swarm is only a scheduler > of container, which is like nova in openstack. Docker compose is a > orchestration program which is like heat in openstack. k8s is the > combination of scheduler and orchestration. So I think it is better > to expose the apis in compose to users which are at the same level as k8s. > Why should the users be deprived of direct access to the Swarm API when it is there? Note also that Compose addresses more general, and differently focused, orchestration than Kubernetes; the latter only offers homogenous scaling groups --- which a docker-compose.yaml file can not even describe. Regards, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Compute API (Was Re: [nova][cinder] how to handle AZ bug 1496235?)
> From: Monty Taylor> To: Sylvain Bauza , "OpenStack Development > Mailing List (not for usage questions)" > Date: 09/28/2015 09:54 AM > Subject: Re: [openstack-dev] Compute API (Was Re: [nova][cinder] how > to handle AZ bug 1496235?) > > ... > Specifically, I want "nova boot" to get me a VM with an IP address. I > don't want it to do fancy orchestration - I want it to not need fancy > orchestration, because needing fancy orchestration to get a VM on a > network is not a feature. > > I also VERY MUCH do not want to need Heat to get a VM. I want to use > Heat to do something complex. Getting a VM is not complex. It should not > be complex. What it's complex and to the level of needing Heat, we've > failed somewhere else. > > Also, people should stop deploying clouds that require people to use > floating IPs to get basic internet access. It's a misuse of the construct. > > Public Network "ext-net" -> shared / directly attachable > Per-tenant Network "private" -> private network, not shared, not routable > > If the user chooses, a router can be added with gateway set to ext-net. > > This way: > > nova boot --network=ext-net -> vm dhcp'd on the public network > nova boot --network=private -> vm dhcp'd on the private network > nova floating-ip-attach -> vm gets a floating ip attached to their > vm from the ext-net network > > All of the use cases are handled, basic things are easy (boot a vm on > the network works in one step) and for the 5% of cases where a floating > IP is actually needed (a long-lived service on a single vm that wants to > keep the IP and not just a DNS name across VM migrations and isn't using > a load-balancer) can use that. > > This is, btw, the most common public cloud deployment model. > > Let's stop making things harder than they need to be and serve our users. As an operator, +1 Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [devstack] Is there a way to configure devstack for one flat external network using Kilo, Neutron?
Is there a way to configure devstack to install Neutron such that there is just one network and that is an external network and Nova can create Compute Instances on it, using projects of Kilo vintage? Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model
Monty Taylorwrote on 09/15/2015 11:04:07 AM: > a) an update to python-novaclient to allow a named network to be passed > to satisfy the "you have more than one network" - the nics argument is > still useful for more complex things I am not using the latest, but rather Juno. I find that in many places the Neutron CLI insists on a UUID when a name could be used. Three cheers for any campaign to fix that. And, yeah, creating VMs on a shared public network is good too. Thanks, mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model
"Armando M."wrote on 09/15/2015 03:50:24 PM: > On 15 September 2015 at 10:02, Doug Hellmann wrote: > Excerpts from Armando M.'s message of 2015-09-15 09:30:35 -0700: ... > As with the Glance image upload API discussion, this is an example > of an extremely common use case that is either complex for the end > user or for which they have to know something about the deployment > in order to do it at all. The usability of an OpenStack cloud running > neutron would be enhanced greatly if there was a simple, clear, way > for the user to get a new VM with a public IP on any cloud without > multiple steps on their part. <> ... > > So this boils down to: in light of the possible ways of providing VM > connectivity, how can we make a choice on the user's behalf? Can we > assume that he/she always want a publicly facing VM connected to > Internet? The answer is 'no'. While it may be true that in some deployments there is no good way for the code to choose, I think that is not the end of the story here. The motivation to do this is that in *some* deployments there *is* a good way for the code to figure out what to do. Regards, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model
Clark Boylanwrote on 09/15/2015 04:06:26 PM: > > I also strongly recommend to users to put vms on a private network and > > use floating ip's/load balancers. For many reasons. Such as, if you > > don't, the ip that gets assigned to the vm helps it become a pet. you > > can't replace the vm and get the same IP. Floating IP's and load > > balancers can help prevent pets. It also prevents security issues with > > DNS and IP's. Also, for every floating ip/lb I have, I usually have 3x or > > more the number of instances that are on the private network. Sure its > > easy to put everything on the public network, but it provides much better > > security if you only put what you must on the public network. Consider > > the internet. would you want to expose every device in your house > > directly on the internet? No. you put them in a private network and poke > > holes just for the stuff that does. we should be encouraging good > > security practices. If we encourage bad ones, then it will bite us later > > when OpenStack gets a reputation for being associated with compromises. > There are a few issues with this. Neutron IPv6 does not support floating > IPs. So now you have to use two completely different concepts for > networking on a single dual stacked VM. IPv4 goes on a private network > and you attach a floating IP. IPv6 is publicly routable. If security and > DNS and not making pets were really the driving force behind floating > IPs we would see IPv6 support them too. These aren't the reasons > floating IPs exist, they exist because we are running out of IPv4 > addresses and NAT is everyones preferred solution to that problem. But > that doesn't make it a good default for a cloud; use them if you are > affected by an IP shortage. > > Nothing prevents you from load balancing against public IPs to address > the DNS and firewall rule concerns (basically don't make pets). This > works great and is how OpenStack's git mirrors work. > > It is also easy to firewall public IPs using Neutron via security groups > (and possibly the firewall service? I have never used it and don't > know). All this to say I think it is reasonable to use public shared > networks by default particularly since IPv6 does not have any concept of > a floating IP in Neutron so using them is just odd unless you really > really need them and you aren't actually any less secure. I'm really glad to see the IPv6 front opened. But I have to say that the analysis of options for securing public addresses omits one case that I think is important: using an external (to Neutron) "appliance". In my environment this is more or less required. This reinforces the bifurcation of addresses that was mentioned: some VMs are private and do not need any service from the external appliance, while others have addresses that need the external appliance on the public/private path. In fact, for this reason, I have taken to using two "external" networks (from Neutron's point of view) --- one whose addresses are handled by the external appliance and one whose addresses are not. In fact, both ranges of address are on the same VLAN. This is FYI, some people have wondered why these thins might be done. > Not to get too off topic, but I would love it if all the devices in my > home were publicly routable. I can use my firewall to punch holes for > them, NAT is not required. Unfortunately I still have issues with IPv6 > at home. Maybe one day this will be a reality :) Frankly, given the propensity for bugs to be discovered, I am glad that nothing in my home is accessible from the outside (aside from the device that does firewall, and I worry about that too). Not that this is really germane to what we want to do for internet-accessible applications/services. Regards, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [fwaas] -IPv6 support in Kilo
From: Rukhsana Ansari rukhsana.ans...@oneconvergence.com To: openstack-dev@lists.openstack.org Date: 06/02/2015 01:59 PM Subject: [openstack-dev] [neutron] [fwaas] -IPv6 support in Kilo Hi, I was browsing the code to understand IPv6 support For FWaaS in Kilo. I don't see a restriction in the db code or in reference fwaas_plugin.py However, from this: https://github.com/openstack/neutron-fwaas/blob/stable/kilo/ neutron_fwaas/services/firewall/drivers/vyatta/vyatta_fwaas.py#L126 I gather that at least Vyatta does not have IPv6 firewall support. Would greatly appreciate it if someone could explain the reasons for this restriction. Thanks -Rukhsana Indeed, this is a surprise to me. http://www.brocade.com/downloads/documents/html_product_manuals/vyatta/vyatta_5400_manual/wwhelp/wwhimpl/js/html/wwhelp.htm#href=Firewall/Firewall_Overview.02.11.html#1749726 indicates that the Vyatta 5400, at least, definitely has firewall functionality. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] IPv4 transition/interoperation with IPv6
Robert Li (baoli) ba...@cisco.com wrote on 05/05/2015 09:02:08 AM: Currently dual stack is supported. Can you be specific on what interoperation/transition techniques you are interested in? We’ve been thinking about NAT64 (stateless or stateful). thanks, Robert On 5/4/15, 9:56 PM, Mike Spreitzer mspre...@us.ibm.com wrote: Does Neutron support any of the 4/6 interoperation/transition techniques? I wear an operator's hat nowadays, and want to make IPv6 as useful and easy to use as possible for my tenants. I think the interoperation/transition techniques will play a big role in this. Is dual stacking working in routers now? At the moment I am still using Juno, but plan to move to Kilo. I want to encourage my tenants to use as much IPv6 as possible. But I expect some will have to keep some of their workload on v4 (I know there is on-going work to get many application frameworks up to v6 speed, and it is not complete yet). I expect some tenants could be mixed: some workload on v4 and some on v6. Such a tenant would appreciate a NAT between his v6 space and his v4 space. This is the easiest cases --- sections 2.5 and 2.6 --- of RFC 6144. I would prefer to do it in a stateless way if possible. That would be pretty easy if Neutron and Nova were willing to accept an IPv6 subnet that is much smaller than 2^64 addresses. I see that my macs differ only in their last 24 bits. Some tenants could put their entire workload on v6, but that workload would be unreachable from customers of all those ISPs (such as mine, CableVision) that deny IPv6 service to their customers. There are techniques for coping, and Teredo looks like a pretty good one. It has been shipped in Windows for years. Yet I can not find a Windows machine where the Teredo actually works. What's up with that? If Windows somehow got its Teredo, or other, act together, that would be only half the job; Teredo requires something from the server side as well, right? Supposing a focus on mobile, where IPv6 is much more available, and/or progress by Microsoft and/or other ISPs, my tenant might face a situation where his clients could come in over v6 but some of his servers still have to run on v4. That's section 2.3 of RFC 6144. While I am a Neutron operator, I am also a customer of a lower layer network provider. That network provider will happily give me a few /64. How do I serve IPv6 subnets to lots of my tenants? In the bad old v4 days this would be easy: a tenant puts all his stuff on his private networks and NATs (e.g., floating IP) his edge servers onto a public network --- no need to align tenant private subnets with public subnets. But with no NAT for v6, there is no public/private distinction --- I can only give out the public v6 subnets that I am given. Yes, NAT is bad. But not being able to get your job done is worse. Sean M. Collins s...@coreitpro.com wrote on 05/05/2015 06:26:28 AM: I think that Neutron exposes enough primitives through the API that advanced services for handling your transition technique of choice could be built. I think that is right, if I am willing to assume Neutron is using OVS --- or build a bunch of alternatives that correspond to all the Neutron plugins and mechanisms that I might encounter. And it would feel a lot like Neutron implementation work. Really, it is one instance of doing some NFV. Thanks, Mike __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] IPv4 transition/interoperation with IPv6
Does Neutron support any of the 4/6 interoperation/transition techniques? I wear an operator's hat nowadays, and want to make IPv6 as useful and easy to use as possible for my tenants. I think the interoperation/transition techniques will play a big role in this. Thanks, Mike__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] multiple external networks on the same host NIC
As an operator, is there a way I can create two external networks from Neutron's point of view, where both of those networks are accessed through the same host NIC? Obviously those networks would be using different subnets. I need this sort of thing because the two subnets are treated differently by the stuff outside of OpenStack, so I need a way that a tenant can get a floating IP of the sort he wants. Since Neutron equates floating IP allocation pools with external networks, I need two external networks. I asked this on openstack-operators, and got one response saying my choices are either to use VLAN tagging on that NIC or manually make more OVS switches that ultimately get their external connectivity through the same NIC. The VLAN tagging is problematic in my environment. Is there any other choice? BTW, I am currently using Juno but plan to move to Kilo. BTW, is Neutron creating any ebtables rules that will get in my way? Thanks, Mike__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Are you looking at scaling the numbers of tenants, Neutron routers, and tenant networks as you scale hosts and guests? I think this is a plausible way to grow. The compartmentalizations that comes with growing those things may make a difference in results. Thanks, Mike From: Neil Jerram neil.jer...@metaswitch.com To: openstack-dev@lists.openstack.org Date: 04/08/2015 12:29 PM Subject:[openstack-dev] [neutron] Neutron scaling datapoints? My team is working on experiments looking at how far the Neutron server will scale, with increasing numbers of compute hosts and VMs. Does anyone have any datapoints on this that they can share? Or any clever hints? I'm already aware of the following ones: https://javacruft.wordpress.com/2014/06/18/168k-instances/ Icehouse 118 compute hosts 80 Neutron server processes (10 per core on each of 8 cores, on the controller node) 27,000 VMs - but only after disabling all security/iptables http://www.opencontrail.org/openstack-neutron-at-scale/ 1000 hosts 5000 VMs 3 Neutron servers (via a load balancer) But doesn't describe if any specific configuration is needed for this. (Other than using OpenContrail! :-)) Many thanks! Neil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Heat: autoscaling across availability zones
From: Weidong Shao weidongs...@gmail.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 02/28/2015 02:12 PM Subject: [openstack-dev] Heat: autoscaling across availability zones From the heat template reference doc, it seems that auto-scaling across AZs is not supported. Is there any blueprint or plan in adding this support? Yes. Here are some ways you can find that work: https://blueprints.launchpad.net/openstack/?searchtext=availabilityzone https://review.openstack.org/#/q/message:availability+message:zone,n,z includes some relevant answers; I have not yet figured out how to narrow the query to match only items in the heat and heat-specs projects. Subscribe to notifications from Gerrit about activity in the Heat project, watch the email subject lines as they whizz by. Regards, Mike__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] AutoScalingGroup with LB Member
Cameron Seader csea...@suse.com wrote on 11/03/2014 06:22:32 PM: Please if you can help me out.. Here is my heat stack http://paste.opensuse.org/80575159 for some reason its failing on the member address.. not sure the syntax is right. Compare with https://github.com/openstack/heat-templates/blob/master/hot/asg_of_stacks.yaml Perhaps the text in the Template Guide ( http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::AutoScalingGroup ) is confusing on this point. The value of the resource property should be the definition of one resource (i.e., the value to which the resource name is mapped in the YAML), not a set of resource definitions. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][ceilometer]: scale up/ down based on number of instances in a group
Daniel Comnea comnea.d...@gmail.com wrote on 10/27/2014 07:16:32 AM: Yes i did but if you look at this example https://github.com/openstack/heat-templates/blob/master/hot/autoscaling.yaml the flow is simple: CPU alarm in Ceilometer triggers the type: OS::Heat::ScalingPolicy which then triggers the type: OS::Heat::AutoScalingGroup Actually the ScalingPolicy does not trigger the ASG. BTW, ScalingPolicy is mis-named; it is not a full policy, it is only an action (the condition part is missing --- as you noted, that is in the Ceilometer alarm). The so-called ScalingPolicy does the action itself when triggered. But it respects your configured min and max size. What are you concerned about making your scaling group smaller than your configured minimum? Just checking here that there is not a misunderstanding. As Clint noted, there is a large-scale effort underway to make Heat maintain what it creates despite deletion of the underlying resources. There is also a small-scale effort underway to make ASGs recover from members stopping proper functioning for whatever reason. See https://review.openstack.org/#/c/127884/ for a proposed interface and initial implementation. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Proposal - add support for Markdown for docs
Sean Dague s...@dague.net wrote on 10/27/2014 09:13:27 AM: From: Sean Dague s...@dague.net On 10/22/2014 11:10 AM, Collins, Sean wrote: With some xargs, sed, and pandoc - I now present to you the first attempt at converting the DevStack docs to RST, and making the doc build look similar to other projects. https://review.openstack.org/130241 It is extremely rough, I basically ran everything through Pandoc and cleaned up any errors that Sphinx spat out. I'm sure there is a lot of work that needs to be done to format it to be more readable - but I'm pretty pleased with the result. +2, you are awesome for taking this on! Thanks much. -Sean Yes, thank you very much --- both for the doc infrastructure and the docs that you will write using it. Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] How can we get more feedback from users?
Angus Salkeld asalk...@mirantis.com wrote on 10/24/2014 12:32:04 AM: I have felt some grumblings about usability issues with Heat templates/client/etc.. and wanted a way that users could come and give us feedback easily (low barrier). I started an etherpad (https:// etherpad.openstack.org/p/heat-useablity-improvements) - the first win is it is spelt wrong :-O We now have some great feedback there in a very short time, most of this we should be able to solve. This lead me to think, should OpenStack have a more general mechanism for users to provide feedback. The idea is this is not for bugs or support, but for users to express pain points, requests for features and docs/howtos. It's not easy to improve your software unless you are listening to your users. Ideas? I very much agree with this. I am actually surprised that OpenStack does not have something fairly formal and organized about this. I suppose it is part of the TC's job, but I think we need more than they can do. I would suggest some sort of user's council that gets involved in blueprint and change reviews. Perhaps after first working toward some degree of consensus and some degree of shaping what the developers work on in each release (this latter is part of the overall program of improving review queue speed, by better focusing developers and reviewers on some shared agenda). Other products have discussion fora, some explicitly dedicated to feedback. You could approximate this with etherpads, or use some real discussion forum platform. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] add cyclomatic complexity check to pep8 target
I like the idea of measuring complexity. I looked briefly at `python -m mccabe`. It seems to measure each method independently. Is this really fair? If I have a class with some big methods, and I break it down into more numerous and smaller methods, then the largest method gets smaller, but the number of methods gets larger. A large number of methods is itself a form of complexity. It is not clear to me that said re-org has necessarily made the class easier to understand. I can also break one class into two, but it is not clear to me that the project has necessarily become easier to understand. While it is true that when you truly make a project easier to understand you sometimes break it into more classes, it is also true that you can do a bad job of re-organizing a set of classes while still reducing the size of the largest method. Has the McCabe metric been evaluated on Python projects? There is a danger in focusing on what is easy to measure if that is not really what you want to optimize. BTW, I find that one of the complexity issues for me when I am learning about a Python class is doing the whole-program type inference so that I know what the arguments are. It seems to me that if you want to measure complexity of Python code then something like the complexity of the argument typing should be taken into account. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [ceilometer] [heat] grace period for an alarm?
Can a Ceilometer client create an alarm with a grace period? That is, an initial period of time during which alarms are suppressed. For a related concept, see the grace period in an AWS autoscaling group. Note that in an OpenStack heat template, the author can not write an arithmetic expression --- any time written must be a literal (and thus apply equally to all members of a scaling group, regardless of when they were created). Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-dev][nova] git fetch fails with permission denied (publickey)
Jennifer Mulsow/Austin/IBM@IBMUS wrote on 10/06/2014 06:45:21 PM: I have followed the instructions on https://wiki.openstack.org/wiki/ Gerrit_Workflow and am trying to fetch the nova repository, but it fails with Permission denied (publickey). I believe this has something to do with my specific account on review.openstack.org, but not sure what. git remote add origin ssh://jmul...@review.openstack.org:29418/ openstack/nova.git git fetch origin Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. Maybe I am blind tonight, but I do not see where the Gerrit_Workflow wiki page is telling you to `git fetch origin`. When starting from scratch (which I suppose you are), you want a command like this: git clone git://git.openstack.org/openstack/nova.git Your SSH test looks valid to me, I do not see why it should fail. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state
Clint Byrum cl...@fewbar.com wrote on 10/01/2014 09:50:33 PM: Recently we've been testing image based updates using TripleO, and we've run into an interesting conundrum. Currently, our image build scripts create a user per service for the image. We don't, at this time, assert a UID, so it could get any UID in the /etc/passwd database of the image. However, if we add a service that happens to have its users created before a previously existing service, the UID's shift by one. When this new image is deployed, the username might be 'ceilometer', but /mnt/state/var/lib/ceilometer is now owned by 'cinder'. I do not understand the problem statement. Unfortunately, I am not familiar with image based updates using TripleO. What is updating what? If the UIDs are not asserted, what UIDs shift by one? Is this because some files keep owner UID while the some UID=name binding in /etc/passwd changes? Or the other way around? Why would there be a change in either? If there is no short answer, don't worry about it. Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM: On 09/24/2014 03:31 PM, Alan Kavanagh wrote: Steven I have to ask what is the motivation and benefits we get from integrating Kubernetes into Openstack? Would be really useful if you can elaborate and outline some use cases and benefits Openstack and Kubernetes can gain. /Alan Alan, I am either unaware or ignorant of another Docker scheduler that is currently available that has a big (100+ folks) development community. Kubernetes meets these requirements and is my main motivation for using it to schedule Docker containers. There are other ways to skin this cat - The TripleO folks wanted at one point to deploy nova with the nova docker VM manager to do such a thing. This model seemed a little clunky to me since it isn't purpose built around containers. Does TripleO require container functionality that is not available when using the Docker driver for Nova? As far as I can tell, the quantitative handling of capacities and demands in Kubernetes is much inferior to what Nova does today. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. If TripleO already knows it wants to run a specific Docker image on a specific host then TripleO does not need a scheduler. These docker containers are then composed of the various config tools and services for each detailed service in OpenStack. For example, mysql would be a container, and tools to configure the mysql service would exist in the container. Kubernetes would pass config options for the mysql database prior to scheduling I am not sure what is meant here by pass config options nor how it would be done prior to scheduling; can you please clarify? I do not imagine Kubernetes would *choose* the config values, K8s does not know anything about configuring OpenStack. Before scheduling, there is no running container to pass anything to. and once scheduled, Kubernetes would be responsible for connecting the various containers together. Kubernetes has a limited role in connecting containers together. K8s creates the networking environment in which the containers *can* communicate, and passes environment variables into containers telling them from what protocol://host:port/ to import each imported endpoint. Kubernetes creates a universal reverse proxy on each minion, to provide endpoints that do not vary as the servers move around. It is up to stuff outside Kubernetes to decide what should be connected to what, and it is up to the containers to read the environment variables and actually connect. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
Clint Byrum cl...@fewbar.com wrote on 09/25/2014 12:13:53 AM: Excerpts from Mike Spreitzer's message of 2014-09-24 20:49:20 -0700: Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM: ... ... Does TripleO require container functionality that is not available when using the Docker driver for Nova? As far as I can tell, the quantitative handling of capacities and demands in Kubernetes is much inferior to what Nova does today. Yes, TripleO needs to manage baremetal and containers from a single host. Nova and Neutron do not offer this as a feature unfortunately. In what sense would Kubernetes manage baremetal (at all)? By from a single host do you mean that a client on one host can manage remote baremetal and containers? I can see that Kubernetes allows a client on one host to get containers placed remotely --- but so does the Docker driver for Nova. As far as use cases go, the main use case is to run a specific Docker container on a specific Kubernetes minion bare metal host. Clint, in another branch of this email tree you referred to the VMs that host Kubernetes. How does that square with Steve's text that seems to imply bare metal minions? I can see that some people have had much more detailed design discussions than I have yet found. Perhaps it would be helpful to share an organized presentation of the design thoughts in more detail. If TripleO already knows it wants to run a specific Docker image on a specific host then TripleO does not need a scheduler. TripleO does not ever specify destination host, because Nova does not allow that, nor should it. It does want to isolate failure domains so that all three Galera nodes aren't on the same PDU, but we've not really gotten to the point where we can do that yet. So I am still not clear on what Steve is trying to say is the main use case. Kubernetes is even farther from balancing among PDUs than Nova is. At least Nova has a framework in which this issue can be posed and solved. I mean a framework that actually can carry the necessary information. The Kubernetes scheduler interface is extremely impoverished in the information it passes and it uses GO structs --- which, like C structs, can not be subclassed. Nova's filter scheduler includes a fatal bug that bites when balancing and you want more than one element per area, see https://bugs.launchpad.net/nova/+bug/1373478. However: (a) you might not need more than one element per area and (b) fixing that bug is a much smaller job than expanding the mind of K8s. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker
I don't know if anyone else has noticed, but you can not install Cinder inside a container. Cinder requires an iSCSI package that fails to install; its install script tries to launch the daemon, and that fails. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] naming of provider template for docs
On further thought, I noticed that template-based resource also describes an AWS::CloudFormation::Stack; and since those are template-based, you could well describe them as custom too. Would you consider nested stack to also describe resources of other types that are implemented by Python code (e.g., for scaling groups) that creates another stack? I think the name we are trying to settle upon means nested stack that is created by supplying, as the resource type seen after mapping through the effective environment, a URL reference to a template. I really think there is no hope of coming up with a reasonable name that includes all of that idea. And in my own work, I have not needed a name that has that specific meaning. I find myself more interested in nested stacks, regardless of the surface details of how they are requested. Yes, the surface details have to be handled in some cases (e.g., balancing a scaling group across AZs), but that is a relatively small and confined matter --- at least in my own work; YMMV. So my new bottom line is this: (1) for the name of the surface syntax details pick any short name that is not too confusing/awful, (2) in the documentation closely bind that name to a explanation that lays out all the critical parts of the idea, and (3) everywhere that you care about nested stacks regardless of the surface details of how they are requested in a template, use the term nested stack. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] naming of provider template for docs
Angus Salkeld asalk...@mirantis.com wrote on 09/18/2014 09:33:56 PM: Hi I am trying to add some docs to openstack-manuals hot_guide about using provider templates : https://review.openstack.org/#/c/121741/ Mike has suggested we use a different term, he thinks provider is confusing. I agree that at the minimum, it is not very descriptive. Mike has suggested nested stack, I personally think this means something a bit more general to many of us (it includes the concept of aws stacks) and may I suggest template resource - note this is even the class name for this exact functionality. Thoughts? Option 1) stay as is provider templates Option 2) nested stack Option 3) template resource Thanks for rising to the documentation challenge and trying to get good terminology. I think your intent is to describe a category of resources, so your option 3 is superior to option 1 --- the thing being described is not a template, it is a resource (made from a template). I think Option 4) custom resource would be even better. My problem with template resource is that, to someone who does not already know what it means, this looks like it might be a kind of resource that is a template (e.g., for consumption by some other resource that does something with a template), rather than itself being something made from a template. If you want to follow this direction to something perfectly clear, you might try templated resource (which is a little better) or template-based resource (which I think is pretty clear but a bit wordy) --- but an AWS::CloudFormation::Stack is also based on a template. I think that if you try for a name that really says all of the critical parts of the idea, you will get something that is too wordy and/or awkward. It is true that custom resource begs the question of how the user accomplishes her customization, but at least now we have the reader asking the right question instead of being misled. I agree that nested stack is a more general concept. It describes the net effect, which the things we are naming have in common with AWS::CloudFormation::Stack. I think it would make sense for our documentation to say something like both an AWS::CloudFormation::Stack and a custom resource are ways to specify a nested stack. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat] Confused about the future of health maintenance and OS::Heat::HARestarter
Background: Health maintenance is very important to users, and I have users who want to do it now and into the future. Today a Heat user can write a template that maintains the health of a resource R. The detection of a health problem can be done by anything that hits a webhook. That generality is important; it is not sufficient to determine health by looking at what physical and/or virtual resources exist, it is also highly desirable to test whether these things are functioning well (e.g., the URL based health checking possible through an OS::Neutron::Pool; e.g., the user has his own external system that detects health problems). The webhook is provided by an OS::Heat::HARestarter (note the name bug: such a thing does not restart anything, rather it deletes and re-creates a given resource and all its dependents) that deletes and re-creates R and its health detection/recovery wiring. For a more specific example, consider the case of detection using the services of an OS::Neutron::Pool. Note that it is not even necessary for there to be workload traffic through the associated OS::Neutron::LoadBalancer; all we are using here is the monitoring prescribed by the Pool's OS::Neutron::HealthMonitor. The user's template has, in addition to R, three things: (1) an OS::Neutron::PoolMember that puts R in the Pool, (2) an OS::Heat::HARestarter that deletes and re-creates R and all its dependents, and (3) a Ceilometer alarm that detects when Neutron is reporting that the PoolMember is unhealthy and responds by hitting the HARestarter's webhook. Note that all three of those are dependent on R, and thus are deleted and re-created when the HARestarter's webhook is hit; this avoids most of the noted issues with HARestarter. R can be a stack that includes both a Nova server and an OS::Neutron::Port, to work around a Nova bug with implicit ports. There is a movement afoot to remove HARestarter. My concern is what can users do, now and into the future. The first and most basic issue is this: at every step in the roadmap, it must be possible for users to accomplish health maintenance. The second issue is easing the impact on what users write. It would be pretty bad if the roadmap looks like this: before point X, users can only accomplish health maintenance as I outlined above, and from point X onward the user has to do something different. That is, there should be a transition period during which users can do things either the old way or the new way. It would be even better if we, or a cloud provider, could provide an abstraction that will be usable throughout the roadmap (once that abstraction becomes available). For example, if there were a resource type OS::Heat::ReliableAutoScalingGroup that adds health maintenance functionality (with detection by an OS::Neutron::Pool and exposure of per-member webhooks usable by anything) to OS::Heat::AutoScalingGroup. Once some other way to do that maintenance becomes available, the implementation of OS::Heat::ReliableAutoScalingGroup could switch to that without requiring any changes to users' templates. If at some point in the future OS::Heat::ReliableAutoScalingGroup becomes exactly equivalent to OS::Heat::AutoScalingGroup then we could deprecate OS::Heat::ReliableAutoScalingGroup and, at a later time, remove it. Even better: since health maintenance is not logically connected to scaling group membership, make the abstraction be simply OS::Heat::HealthyResource (i.e., it is about a single resource regardless of whether it is a member of a scaling group) rather than OS::Heat::ReliableAutoScalingGroup. Question: would that abstraction (including the higher level detection and exposure of re-creation webhook) be implementable (or a no-op) in the planned future? To aid in understanding: while it may be distasteful for a resource like HARestarter to tweak its containing stack, the critical question is whether it will remain *possible* throughout a transition period. Is there an issue with such hacks being *possible* throughout a reasonable transition period? Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] referencing the index of a ResourceGroup
Steven Hardy sha...@redhat.com wrote on 09/11/2014 04:21:18 AM: On Wed, Sep 10, 2014 at 04:44:01PM -0500, Jason Greathouse wrote: I'm trying to find a way to create a set of servers and attach a new volume to each server. ... Basically creating lots of resource groups for related things is the wrong pattern. You need to create one nested stack template containing the related things (Server, Volume and VolumeAttachment in this case), and use ResourceGroup to multiply them as a unit. I answered a similar question here on the openstack general ML recently (which for future reference may be a better ML for usage questions like this, as it's not really development discussion): http://lists.openstack.org/pipermail/openstack/2014-September/009216.html Here's another example which I used in a summit demo, which I think basically does what you need? https://github.com/hardys/demo_templates/tree/master/ juno_summit_intro_to_heat/example3_server_with_volume_group There is also an example of exactly this under review. See https://review.openstack.org/#/c/97366/ Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat] Re: Change in openstack/heat[master]: Implement AZ spanning for ASGs
You offered to share ideas about a different way to approach spanning AZs for OS::Heat::AutoScalingGroup. I am interested. Can we discuss it here? Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat] heat.conf.sample is not up to date
What is going on with this? If I do a fresh clone of heat and run `tox -epep8` then I get that complaint. If I then run the recommended command to fix it, and then `tox -epep8` again, I get the same complaint again --- and with different differences exhibited! The email below carries a typescript showing this. What I really need to know is what to do when committing a change that really does require a change in the sample configuration file. Of course I tried running generate_sample.sh, but `tox -epep8` still complains. What is the right procedure to get a correct sample committed? BTW, I am doing the following admittedly risky thing: I run DevStack, and make my changes in /opt/stack/heat/. Thanks, Mike - Forwarded by Mike Spreitzer/Watson/IBM on 08/24/2014 03:03 AM - From: ubuntu@mjs-dstk-821a (Ubuntu) To: Mike Spreitzer/Watson/IBM@IBMUS, Date: 08/24/2014 02:55 AM Subject:fresh flake fail ubuntu@mjs-dstk-821a:~/code$ git clone git://git.openstack.org/openstack/heat.git Cloning into 'heat'... remote: Counting objects: 49690, done. remote: Compressing objects: 100% (19765/19765), done. remote: Total 49690 (delta 36660), reused 39014 (delta 26526) Receiving objects: 100% (49690/49690), 7.92 MiB | 7.29 MiB/s, done. Resolving deltas: 100% (36660/36660), done. Checking connectivity... done. ubuntu@mjs-dstk-821a:~/code$ cd heat ubuntu@mjs-dstk-821a:~/code/heat$ tox -epep8 pep8 create: /home/ubuntu/code/heat/.tox/pep8 pep8 installdeps: -r/home/ubuntu/code/heat/requirements.txt, -r/home/ubuntu/code/heat/test-requirements.txt pep8 develop-inst: /home/ubuntu/code/heat pep8 runtests: PYTHONHASHSEED='0' pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn bin/heat-api-cloudwatch bin/heat-engine bin/heat-manage contrib pep8 runtests: commands[1] | /home/ubuntu/code/heat/tools/config/check_uptodate.sh --- /tmp/heat.ep2CBe/heat.conf.sample2014-08-24 06:52:54.16484 + +++ etc/heat/heat.conf.sample2014-08-24 06:48:13.66484 + @@ -164,7 +164,7 @@ #allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions # Qpid broker hostname. (string value) -#qpid_hostname=heat +#qpid_hostname=localhost # Qpid broker port. (integer value) #qpid_port=5672 @@ -221,7 +221,7 @@ # The RabbitMQ broker address where a single node is used. # (string value) -#rabbit_host=heat +#rabbit_host=localhost # The RabbitMQ broker port where a single node is used. # (integer value) check_uptodate.sh: heat.conf.sample is not up to date. check_uptodate.sh: Please run /home/ubuntu/code/heat/tools/config/generate_sample.sh. ERROR: InvocationError: '/home/ubuntu/code/heat/tools/config/check_uptodate.sh' pep8 runtests: commands[2] | /home/ubuntu/code/heat/tools/requirements_style_check.sh requirements.txt test-requirements.txt pep8 runtests: commands[3] | bash -c find heat -type f -regex '.*\.pot?' -print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null ___ summary ERROR: pep8: commands failed ubuntu@mjs-dstk-821a:~/code/heat$ ubuntu@mjs-dstk-821a:~/code/heat$ ubuntu@mjs-dstk-821a:~/code/heat$ tools/config/generate_sample.sh ubuntu@mjs-dstk-821a:~/code/heat$ ubuntu@mjs-dstk-821a:~/code/heat$ ubuntu@mjs-dstk-821a:~/code/heat$ ubuntu@mjs-dstk-821a:~/code/heat$ tox -epep8 pep8 develop-inst-noop: /home/ubuntu/code/heat pep8 runtests: PYTHONHASHSEED='0' pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn bin/heat-api-cloudwatch bin/heat-engine bin/heat-manage contrib pep8 runtests: commands[1] | /home/ubuntu/code/heat/tools/config/check_uptodate.sh --- /tmp/heat.DqIhK5/heat.conf.sample2014-08-24 06:54:34.62884 + +++ etc/heat/heat.conf.sample2014-08-24 06:53:51.54084 + @@ -159,10 +159,6 @@ # Size of RPC connection pool. (integer value) #rpc_conn_pool_size=30 -# Modules of exceptions that are permitted to be recreated -# upon receiving exception data from an rpc call. (list value) -#allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions - # Qpid broker hostname. (string value) #qpid_hostname=heat @@ -301,15 +297,6 @@ # Heartbeat time-to-live. (integer value) #matchmaker_heartbeat_ttl=600 -# Host to locate redis. (string value) -#host=127.0.0.1 - -# Use this port to connect to redis host. (integer value) -#port=6379 - -# Password for Redis server (optional). (string value) -#password=None - # Size of RPC greenthread pool. (integer value) #rpc_thread_pool_size=64 @@ -1229,6 +1216,22 @@ #hash_algorithms=md5 +[matchmaker_redis] + +# +# Options defined in oslo.messaging +# + +# Host to locate redis. (string value) +#host=127.0.0.1 + +# Use this port to connect to redis host. (integer value) +#port=6379 + +# Password for Redis server (optional). (string value) +#password=None + + [matchmaker_ring] # check_uptodate.sh: heat.conf.sample is not up to date
Re: [openstack-dev] [heat] heat.conf.sample is not up to date
The differences do not look to me like ones that would follow from a difference in hash seed. Clint Byrum cl...@fewbar.com wrote on 08/24/2014 05:00:21 AM: From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org, Date: 08/24/2014 05:02 AM Subject: Re: [openstack-dev] [heat] heat.conf.sample is not up to date Guessing this is due to the new tox feature which randomizes python's hash seed. Excerpts from Mike Spreitzer's message of 2014-08-24 00:10:42 -0700: What is going on with this? If I do a fresh clone of heat and run `tox -epep8` then I get that complaint. If I then run the recommended command to fix it, and then `tox -epep8` again, I get the same complaint again --- and with different differences exhibited! The email below carries a typescript showing this. What I really need to know is what to do when committing a change that really does require a change in the sample configuration file. Of course I tried running generate_sample.sh, but `tox -epep8` still complains. What is the right procedure to get a correct sample committed? BTW, I am doing the following admittedly risky thing: I run DevStack, and make my changes in /opt/stack/heat/. Thanks, Mike - Forwarded by Mike Spreitzer/Watson/IBM on 08/24/2014 03:03 AM - From: ubuntu@mjs-dstk-821a (Ubuntu) To: Mike Spreitzer/Watson/IBM@IBMUS, Date: 08/24/2014 02:55 AM Subject:fresh flake fail ubuntu@mjs-dstk-821a:~/code$ git clone git://git.openstack.org/openstack/heat.git Cloning into 'heat'... remote: Counting objects: 49690, done. remote: Compressing objects: 100% (19765/19765), done. remote: Total 49690 (delta 36660), reused 39014 (delta 26526) Receiving objects: 100% (49690/49690), 7.92 MiB | 7.29 MiB/s, done. Resolving deltas: 100% (36660/36660), done. Checking connectivity... done. ubuntu@mjs-dstk-821a:~/code$ cd heat ubuntu@mjs-dstk-821a:~/code/heat$ tox -epep8 pep8 create: /home/ubuntu/code/heat/.tox/pep8 pep8 installdeps: -r/home/ubuntu/code/heat/requirements.txt, -r/home/ubuntu/code/heat/test-requirements.txt pep8 develop-inst: /home/ubuntu/code/heat pep8 runtests: PYTHONHASHSEED='0' pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn bin/heat-api-cloudwatch bin/heat-engine bin/heat-manage contrib pep8 runtests: commands[1] | /home/ubuntu/code/heat/tools/config/check_uptodate.sh --- /tmp/heat.ep2CBe/heat.conf.sample2014-08-24 06:52:54.16484 + +++ etc/heat/heat.conf.sample2014-08-24 06:48:13.66484 + @@ -164,7 +164,7 @@ #allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions # Qpid broker hostname. (string value) -#qpid_hostname=heat +#qpid_hostname=localhost # Qpid broker port. (integer value) #qpid_port=5672 @@ -221,7 +221,7 @@ # The RabbitMQ broker address where a single node is used. # (string value) -#rabbit_host=heat +#rabbit_host=localhost # The RabbitMQ broker port where a single node is used. # (integer value) check_uptodate.sh: heat.conf.sample is not up to date. check_uptodate.sh: Please run /home/ubuntu/code/heat/tools/config/generate_sample.sh. ERROR: InvocationError: '/home/ubuntu/code/heat/tools/config/check_uptodate.sh' pep8 runtests: commands[2] | /home/ubuntu/code/heat/tools/requirements_style_check.sh requirements.txt test-requirements.txt pep8 runtests: commands[3] | bash -c find heat -type f -regex '.*\.pot?' -print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null ___ summary ERROR: pep8: commands failed ubuntu@mjs-dstk-821a:~/code/heat$ ubuntu@mjs-dstk-821a:~/code/heat$ ubuntu@mjs-dstk-821a:~/code/heat$ tools/config/generate_sample.sh ubuntu@mjs-dstk-821a:~/code/heat$ ubuntu@mjs-dstk-821a:~/code/heat$ ubuntu@mjs-dstk-821a:~/code/heat$ ubuntu@mjs-dstk-821a:~/code/heat$ tox -epep8 pep8 develop-inst-noop: /home/ubuntu/code/heat pep8 runtests: PYTHONHASHSEED='0' pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn bin/heat-api-cloudwatch bin/heat-engine bin/heat-manage contrib pep8 runtests: commands[1] | /home/ubuntu/code/heat/tools/config/check_uptodate.sh --- /tmp/heat.DqIhK5/heat.conf.sample2014-08-24 06:54:34.62884 + +++ etc/heat/heat.conf.sample2014-08-24 06:53:51.54084 + @@ -159,10 +159,6 @@ # Size of RPC connection pool. (integer value) #rpc_conn_pool_size=30 -# Modules of exceptions that are permitted to be recreated -# upon receiving exception data from an rpc call. (list value) - #allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions - # Qpid broker hostname. (string value) #qpid_hostname=heat @@ -301,15 +297,6 @@ # Heartbeat time-to-live
Re: [openstack-dev] [heat] heat.conf.sample is not up to date
Morgan Fainberg morgan.fainb...@gmail.com wrote on 08/24/2014 12:01:37 PM: ... Keystone saw an oddity with the new sample config generator (changing how options are sorted and therefore changing the way the sample config is rendered). This could be a similar / related issue. Most of the projects stopped gating on up-to-date sample config a few reasons, the first is that with external library dependencies you never know when / if something upstream will break the test run (e.g. New Oslo.config or new keystonemiddleware). Now imagine that issue occurred and was blocking a gate-fixing bug (happened at least a couple times). In short, making sample config being up-to-date to merge code causes a lot if headaches. Different projects handle this differently. Nova doesn't have a sample config in tree, keystone updates on a semi-regular basis (sometimes as part of a patch, sometimes as a separate patch). Keystone team is looking at adding a simple non-voting gate job (if infra doesn't mind) that will tell us the config is out of date. While it is nice to always have an updated sample config, I think it is not worth the breakage / issues it adds to the gate. It might make sense to standardize how we handle sample config files across the projects or at least standardize on removing the gate block if the config is out of date. I know it was floated earlier that there would be a proposal bot job (like translations) for sample config files, but I don't remember the specifics of why it wasn't well liked. Consistency would be nice. The must-have is just to document each project's procedure (accurately, of course). Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] heat.conf.sample is not up to date
Ruslan Kamaldinov rkamaldi...@mirantis.com wrote on 08/24/2014 12:30:04 PM: From: Ruslan Kamaldinov rkamaldi...@mirantis.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org, Date: 08/24/2014 12:32 PM Subject: Re: [openstack-dev] [heat] heat.conf.sample is not up to date On Sun, Aug 24, 2014 at 11:10 AM, Mike Spreitzer mspre...@us.ibm.com wrote: What I really need to know is what to do when committing a change that really does require a change in the sample configuration file. Of course I tried running generate_sample.sh, but `tox -epep8` still complains. What is the right procedure to get a correct sample committed? BTW, I am doing the following admittedly risky thing: I run DevStack, and make my changes in /opt/stack/heat/. Mike, It seems that you have oslo.messaging installed from master (default behavior of Devstack), while heat.sample.config is built for oslo.messaging v 1.3.1. As a quick workaround I'd suggest to downgrade oslo.messaging to 1.3.1 in pep8 virtual environment: $ cd /opt/stack/heat $ source .tox/pep8/bin/activate $ pip install oslo.messaging==1.3.1 $ deactivate $ tox -e pep8 # should pass now In that same VM in which I ran DevStack, I also made an independent clone of heat in ~/code/heat; my original email quoted a failure from that clone, hoping that it might be easier to debug (being a simpler scenario). Below is a typescript showing me try again there, including a trial of what Mathieu Gagné suggested (which has some kind of command line syntax error, and did nothing). You will see that I investigated and found that in this case tox's venv contained oslo.messaging version 1.3.1, so no adjustment about that was needed. Yet it still failed. ubuntu@mjs-dstk-821a:~/code/heat$ rm -rf .tox ubuntu@mjs-dstk-821a:~/code/heat$ tox -evenv bash ./tools/config/generate_sample.sh -b . -p heat -o etc/heat usage: tox [-h] [--version] [-v] [--showconfig] [-l] [-c CONFIGFILE] [-e envlist] [--notest] [--sdistonly] [--installpkg PATH] [--develop] [--set-home] [-i URL] [-r] [--result-json PATH] [--hashseed SEED] [--force-dep REQ] [--sitepackages] [--skip-missing-interpreters] [args [args ...]] tox: error: unrecognized arguments: -b . -p heat -o etc/heat ubuntu@mjs-dstk-821a:~/code/heat$ tox -epep8 pep8 create: /home/ubuntu/code/heat/.tox/pep8 pep8 installdeps: -r/home/ubuntu/code/heat/requirements.txt, -r/home/ubuntu/code/heat/test-requirements.txt pep8 develop-inst: /home/ubuntu/code/heat pep8 runtests: PYTHONHASHSEED='0' pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn bin/heat-api-cloudwatch bin/heat-engine bin/heat-manage contrib pep8 runtests: commands[1] | /home/ubuntu/code/heat/tools/config/check_uptodate.sh --- /tmp/heat.UnHAZD/heat.conf.sample 2014-08-25 04:06:07.64884 + +++ etc/heat/heat.conf.sample 2014-08-24 06:53:51.54084 + @@ -159,10 +159,6 @@ # Size of RPC connection pool. (integer value) #rpc_conn_pool_size=30 -# Modules of exceptions that are permitted to be recreated -# upon receiving exception data from an rpc call. (list value) -#allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions - # Qpid broker hostname. (string value) #qpid_hostname=heat @@ -301,15 +297,6 @@ # Heartbeat time-to-live. (integer value) #matchmaker_heartbeat_ttl=600 -# Host to locate redis. (string value) -#host=127.0.0.1 - -# Use this port to connect to redis host. (integer value) -#port=6379 - -# Password for Redis server (optional). (string value) -#password=None - # Size of RPC greenthread pool. (integer value) #rpc_thread_pool_size=64 @@ -1229,6 +1216,22 @@ #hash_algorithms=md5 +[matchmaker_redis] + +# +# Options defined in oslo.messaging +# + +# Host to locate redis. (string value) +#host=127.0.0.1 + +# Use this port to connect to redis host. (integer value) +#port=6379 + +# Password for Redis server (optional). (string value) +#password=None + + [matchmaker_ring] # check_uptodate.sh: heat.conf.sample is not up to date. check_uptodate.sh: Please run /home/ubuntu/code/heat/tools/config/generate_sample.sh. ERROR: InvocationError: '/home/ubuntu/code/heat/tools/config/check_uptodate.sh' pep8 runtests: commands[2] | /home/ubuntu/code/heat/tools/requirements_style_check.sh requirements.txt test-requirements.txt pep8 runtests: commands[3] | bash -c find heat -type f -regex '.*\.pot?' -print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null _ __ summary _ ___ ERROR: pep8: commands failed ubuntu@mjs-dstk-821a:~/code/heat$ . .tox/pep8/bin/activate (pep8)ubuntu@mjs-dstk-821a:~/code/heat$ python Python 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] on linux2 Type help, copyright, credits or license for more information. system Traceback (most recent call last): File
Re: [openstack-dev] [OpenStack][Docker] Run OpenStack Service in Docker Container
In particular, I tried to run DevStack inside an LXC a few months ago. I discovered that DevStack (presumably for the sake of cinder-volume) pre-reqs a system package named tgt, and tgt does not succeed to install inside an LXC (the install script launches the daemon, but the daemon launch fails). Regards, Mike From: Jyoti Ranjan jran...@gmail.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org, Date: 08/18/2014 08:53 AM Subject:Re: [openstack-dev] [OpenStack][Docker] Run OpenStack Service in Docker Container I believe that everything can not go as a dock container. For e.g. 1. compute nodes 2. baremetal provisioning 3. L3 router etc My understanding is that container is good mechanism to deploy api-controller and scheduler for many services. For backend component of services (like nova-compute, cinder-volume if LVM is used), I that usage of baremetal is more appropriate (except backend component like cinder-volume for external devices, nova-compute proxy etc). Just thought to check your opinion about my understanding! Need your views! On Mon, Aug 18, 2014 at 3:34 PM, Jay Lau jay.lau@gmail.com wrote: I see that there are some openstack docker images in public docker repo, perhaps you can check them on github to see how to use them. [root@db03b04 ~]# docker search openstack NAME DESCRIPTION STARS OFFICIAL AUTOMATED ewindisch/dockenstackOpenStack development environment (using D... 6[OK] jyidiego/openstack-clientAn ubuntu 12.10 LTS image that has nova, s... 1 dkuffner/docker-openstack-stress A docker container for openstack which pro... 0[OK] garland/docker-openstack-keystone 0[OK] mpaone/openstack 0 nirmata/openstack-base 0 balle/openstack-ipython2-client Features Python 2.7.5, Ipython 2.1.0 and H... 0 booleancandy/openstack_clients 0[OK] leseb/openstack-keystone 0 raxcloud/openstack-client 0 paulczar/openstack-agent 0 booleancandy/openstack-clients 0 jyidiego/openstack-client-rumm-ansible 0 bodenr/jumpgate SoftLayer Jumpgate WSGi OpenStack REST API... 0[OK] sebasmagri/docker-marconiDocker images for the Marconi Message Queu... 0[OK] chamerling/openstack-client 0[OK] centurylink/openstack-cli-wetty This image provides a Wetty terminal with ... 0[OK] 2014-08-18 16:47 GMT+08:00 Philip Cheong philip.che...@elastx.se: I think it's a very interesting test for docker. I too have been think about this for some time to try and dockerise OpenStack services, but as the usual story goes, I have plenty things I'd love to try, but there are only so many hours in a day... Would definitely be interested to hear if anyone has attempted this and what the outcome was. Any suggestions on what the most appropriate service would be to begin with? On 14 August 2014 14:54, Jay Lau jay.lau@gmail.com wrote: I see a few mentions of OpenStack services themselves being containerized in Docker. Is this a serious trend in the community? http://allthingsopen.com/2014/02/12/why-containers-for-openstack-services/ -- Thanks, Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Philip Cheong Elastx | Public and Private PaaS email: philip.che...@elastx.se office: +46 8 557 728 10 mobile: +46 702 8170 814 twitter: @Elastx http://elastx.se ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack][Docker] Run OpenStack Service in Docker Container
Jay Lau jay.lau@gmail.com wrote on 08/14/2014 08:54:56 AM: I see a few mentions of OpenStack services themselves being containerized in Docker. Is this a serious trend in the community? http://allthingsopen.com/2014/02/12/why-containers-for-openstack-services/ It looks to me like the problem addressed there is managing port assignments. To make progress on that you would need something that lets the clients know the hostport endpoints where the servers are found, and you would need both the clients and servers using it. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Separating the issues around smarter/solver/joint/holistic placement
There has been a lot of discussion around these issues, let me see if I can break it down into pieces, hopefully in a way that allows some progress on one of them first. I continue to focus on the timeless version of the problem, in which the placement question is simply where can we put some guest(s) now without making plans for whether/how things may change in the future. There is also a rich discussion to have about placement over time (in other communities this is what scheduling means), but that is not my goal here. I see essentially three issues: (i1) more sophisticated placement criteria, (i2) joint vs sequential decision making, and (i3) single-service vs cross service. I think we could make a roadmap for addressing them in this order. Do you think the issues are separable and attackable in this way? Objections have been raised regarding complexity, and I would like to be clear about what complexity we are discussing. For each of these three issues, I can think of five kinds of complexity to discuss: the development complexity faced by us OpenStack developers and/or a cloud provider who wants to add such things, the runtime costs, the API complexity for allowing the issue to be addressed, the development complexity faced by the cloud users in writing their placement problems, and the size of the written problems. (i1) More sophisticated placement criteria. For example, I think cloud providers would like to have a way to express a preference for consolidation, to help them conserve energy. That's a fairly rich topic in itself, and I hope the details need not concern us here. OTOH, cloud users may want to have a way to express a preference for spreading their guests out --- for example, spreading around within an availability zone. This may seem relatively boring for stateless web servers, but if you think about things like Zookeeper or Cassandra servers you see a serious preference for getting as much failure independence as you can get within a given AZ. Other examples of what cloud users may want cross services, like placing VMs and storage volumes in close proximity (by some measure). Let's consider a couple of concrete examples from the record. One is referenced in the current revision of https://review.openstack.org/#/c/96543/ - namely, http://goo.gl/vuHcz2 . There Yathi exhibits a binary integer programming problem for placing a group of guests so as to minimize the sum of (host's free RAM before any members of this group are placed). Minimizing that objective has a roughly consolidating effect in many situations, even though it does not directly express exactly consolidation (exactly expressing consolidation requires a non-linear objective). Similarly, maximizing that objective has something of a tendency to spread things out, even though the objective does not exactly express spreading. (BTW, I do not think we should assume that all hosts in an AZ have the same capacity vector.) Another example is at https://wiki.openstack.org/wiki/Heat/PolicyExtension#LLMN_Anti-Collocation . There we see a criterion that is a precise spreading condition, for spreading a group of guests to a certain precise degree. (BTW, I tend to use collocation and anti-collocation when speaking of affinity stated in terms of location --- as opposed, for example, to network proximity.) For any collection of more or less sophisticated placement criteria, if we consider the question of placing a single guest --- supposing it is stipulated that some guests have already had their placement determined and others will be determined later --- the question boils down to the exactly the terms that we have in Nova's FilterScheduler today: (a) which hosts can host this guest and (b) ranking the acceptable hosts. Of course, that is not the only way to address a problem of placing several guests --- but it is one valid way, even if the placement problem is stated as a mixed integer programming problem. Group placement problems are NP hard, so solving them exactly is out of the question. The only question is how hard are we going to work at making a good approximation; this is what (i2) is about, and we will get to that below. Let us review the two concrete examples with an eye towards the five kinds of complexity. Let us start with Yathi's RAM minimization binary integer programming example. There is a difficulty if we try to work that example in today's framework. The objective is a function of the amount of RAM that was free on each host *before any group members are placed*. Today's FilterScheduler works on one guest at a time, updating host states as it goes along; it does not offer earlier host states for use in later decisions. However, today we can solve a slightly different problem --- which happens to be an even better approximation of a consolidation problem. That is to minimize the sum of (host's available RAM just before the
Re: [openstack-dev] [nova] Separating the issues around smarter/solver/joint/holistic placement
This is primarily an issue for Nova. Mike Spreitzer/Watson/IBM@IBMUS wrote on 08/20/2014 01:21:24 AM: From: Mike Spreitzer/Watson/IBM@IBMUS To: OpenStack Development Mailing List openstack-dev@lists.openstack.org, Date: 08/20/2014 01:24 AM Subject: [openstack-dev] Separating the issues around smarter/ solver/joint/holistic placement There has been a lot of discussion around these issues, let me see if I can break it down into pieces, hopefully in a way that allows some progress on one of them first. I continue to focus on the timeless version of the problem, in which the placement question is simply where can we put some guest(s) now without making plans for whether/how things may change in the future. There is also a rich discussion to have about placement over time (in other communities this is what scheduling means), but that is not my goal here. I see essentially three issues: (i1) more sophisticated placement criteria, (i2) joint vs sequential decision making, and (i3) single- service vs cross service. I think we could make a roadmap for addressing them in this order. Do you think the issues are separable and attackable in this way? Objections have been raised regarding complexity, and I would like to be clear about what complexity we are discussing. For each of these three issues, I can think of five kinds of complexity to discuss: the development complexity faced by us OpenStack developers and/or a cloud provider who wants to add such things, the runtime costs, the API complexity for allowing the issue to be addressed, the development complexity faced by the cloud users in writing their placement problems, and the size of the written problems. (i1) More sophisticated placement criteria. For example, I think cloud providers would like to have a way to express a preference for consolidation, to help them conserve energy. That's a fairly rich topic in itself, and I hope the details need not concern us here. OTOH, cloud users may want to have a way to express a preference for spreading their guests out --- for example, spreading around within an availability zone. This may seem relatively boring for stateless web servers, but if you think about things like Zookeeper or Cassandra servers you see a serious preference for getting as much failure independence as you can get within a given AZ. Other examples of what cloud users may want cross services, like placing VMs and storage volumes in close proximity (by some measure). Let's consider a couple of concrete examples from the record. One is referenced in the current revision of https:// review.openstack.org/#/c/96543/ - namely, http://goo.gl/vuHcz2 . There Yathi exhibits a binary integer programming problem for placing a group of guests so as to minimize the sum of (host's free RAM before any members of this group are placed). Minimizing that objective has a roughly consolidating effect in many situations, even though it does not directly express exactly consolidation (exactly expressing consolidation requires a non-linear objective). Similarly, maximizing that objective has something of a tendency to spread things out, even though the objective does not exactly express spreading. (BTW, I do not think we should assume that all hosts in an AZ have the same capacity vector.) Another example is at https://wiki.openstack.org/wiki/Heat/ PolicyExtension#LLMN_Anti-Collocation . There we see a criterion that is a precise spreading condition, for spreading a group of guests to a certain precise degree. (BTW, I tend to use collocation and anti-collocation when speaking of affinity stated in terms of location --- as opposed, for example, to network proximity.) For any collection of more or less sophisticated placement criteria, if we consider the question of placing a single guest --- supposing it is stipulated that some guests have already had their placement determined and others will be determined later --- the question boils down to the exactly the terms that we have in Nova's FilterScheduler today: (a) which hosts can host this guest and (b) ranking the acceptable hosts. Of course, that is not the only way to address a problem of placing several guests --- but it is one valid way, even if the placement problem is stated as a mixed integer programming problem. Group placement problems are NP hard, so solving them exactly is out of the question. The only question is how hard are we going to work at making a good approximation; this is what (i2) is about, and we will get to that below. Let us review the two concrete examples with an eye towards the five kinds of complexity. Let us start with Yathi's RAM minimization binary integer programming example. There is a difficulty if we try to work that example in today's framework. The objective is a function of the amount of RAM that was free
Re: [openstack-dev] OS or os are not acronyms for OpenStack
Anita Kuno ante...@anteaya.info wrote on 08/15/2014 10:38:20 AM: OpenStack is OpenStack. The use of openstack is also acceptable in our development conversations. OS or os is operating system. I am starting to see some people us OS or os to mean OpenStack. This is confusing and also incorrect[0]. ... I have seen OS for OpenStack from the start. Just look at the environment variables that the CLI reads. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OS or os are not acronyms for OpenStack
Anita Kuno ante...@anteaya.info wrote on 08/15/2014 01:08:44 PM: ... I think you hit the nail on the head here, Russell, it's fine in the right context. The definition of the right context however is somewhat elusive. I have chosen (it is my own fault) to place myself in the area where the folks I deal with struggle with understanding context. The newcomers to the third party space and folks creating stackforge repos don't have the benefit of the understanding that core reviewers have (would I be accurate in saying that it is mostly nova reviewers who have responded to my initial post thus far?). I suffered from an instance of this confusion myself when I was just getting started, and have seen colleagues get confused too. I suspect this problem hits many newcomers. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OS or os are not acronyms for OpenStack
Russell Bryant rbry...@redhat.com wrote on 08/15/2014 01:49:40 PM: ... but surely when it comes to learning OpenStack itself, the OpenStack community, dev processes, tools, etc this has got to be extremely far down the list of barriers to entry. No argument there. I am spending decimal orders of magnitude more time looking for a working concrete example of using DevStack to install OpenStack with Neutron inside a VM so that the nested VMs can talk to the outside world. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron][DevStack] How to increase developer usage of Neutron
I'll bet I am not the only developer who is not highly competent with bridges and tunnels, Open VSwitch, Neutron configuration, and how DevStack transmutes all those. My bet is that you would have more developers using Neutron if there were an easy-to-find and easy-to-follow recipe to use, to create a developer install of OpenStack with Neutron. One that's a pretty basic and easy case. Let's say a developer gets a recent image of Ubuntu 14.04 from Canonical, and creates an instance in some undercloud, and that instance has just one NIC, at 10.9.8.7/16. If there were a recipe for such a developer to follow from that point on, it would be great. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][DevStack] How to increase developer usage of Neutron
CARVER, PAUL pc2...@att.com wrote on 08/14/2014 09:35:17 AM: Mike Spreitzer [mailto:mspre...@us.ibm.com] wrote: I'll bet I am not the only developer who is not highly competent with bridges and tunnels, Open VSwitch, Neutron configuration, and how DevStack transmutes all those. My bet is that you would have more developers using Neutron if there were an easy-to-find and easy-to-follow recipe to use, to create a developer install of OpenStack with Neutron. One that's a pretty basic and easy case. Let's say a developer gets a recent image of Ubuntu 14.04 from Canonical, and creates an instance in some undercloud, and that instance has just one NIC, at 10.9.8.7/16. If there were a recipe for such a developer to follow from that point on, it would be great. https://wiki.openstack.org/wiki/NeutronDevstack worked for me. However, I'm pretty sure it's only a single node all in one setup. At least, I created only one VM to run it on and I don't think DevStack has created multiple nested VMs inside of the one I create to run DevStack. I haven't gotten around to figuring out how to setup a full multi-node DevStack setup with separate compute nodes and network nodes and GRE/VXLAN tunnels. There are multi-node instructions on that wiki page but I haven't tried following them. If someone has a Vagrant file that creates a full multi- node Neutron devstack complete with GRE/VXLAN tunnels it would be great if they could add it to that wiki page. A working concrete recipe for a single-node install would be great. https://wiki.openstack.org/wiki/NeutronDevstack is far from a concrete recipe, leaving many blanks to be filled in by the reader. My problem is that as a non-expert in the relevant networking arcana, Neutron implementation, and DevStack configuration options, it is not entirely obvious how to fill in the blanks. For starters, http://docs.openstack.org/admin-guide-cloud/content/network-connectivity.html speaks of four networks and, appropriately for a general page like that, does not relate them to NICs. But at the start of the day, I need to know how many NICs to put on my host VM (the one in which I will run DevStack to install OpenStack), how to configure them in the host VM's operating system, and how to tell DevStack whatever details it needs and cannot figure out on its own (I am not even clear on what that set is). I need to know how to derive the fixed and floating IP address ranges from the networking context of my host VM. A recipe that requires more than one NIC on my host VM can be problematic in some situations, which is why I suggested starting with a recipe for a host with a single NIC. I am not using Xen in my undercloud. I suspect many developers are not using Xen. I did not even know it was possible to install OpenStack inside a Xen VM; does that still work? I was hoping for a working concrete recipe that does not depend on the undercloud, rather something that works in a vanilla context that can easily be established with whatever undercloud a given developer is using. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Neutron][Technical Committee] nova-network - Neutron. Throwing a wrench in the Neutron gap analysis
Monty Taylor mord...@inaugust.com wrote on 08/05/2014 12:27:14 PM: On 08/05/2014 09:18 AM, Jay Pipes wrote: Hello stackers, TC, Neutron contributors, At the Nova mid-cycle meetup last week in Oregon, during the discussion about the future of nova-network, the topic of nova-network - Neutron migration came up. For some reason, I had been clueless about the details of one of the items in the gap analysis the TC had requested [1]. Namely, the 5th item, about nova-network - Neutron migration, which is detailed in the following specification: https://review.openstack.org/#/c/101921/12/specs/juno/neutron-migration.rst ... I personally believe that this requirement to support a live migration with no downtime of running instances between a nova-network and a Neutron deployment *is neither realistic, nor worth the extensive time and technical debt needed to make this happen*. I suggest that it would be better to instead provide good instructions for doing cold migration (snapshot VMs in old nova-network deployment, store in Swift or something, then launch VM from a snapshot in new Neutron deployment) -- which should cover the majority of deployments -- and then write some instructions for what to look out for when doing a custom migration for environments that simply cannot afford any downtime and *really* want to migrate to Neutron. For these deployments, it's almost guaranteed that they will need to mangle their existing databases and do manual data migration anyway -- like RAX did when moving from nova-network to Neutron. The variables are too many to list here, and the number of deployments actually *needing* this work seems to me to be very limited. Someone suggested Metacloud *might* be the only deployment that might meet the needs for a live nova-network - Neutron migration. Metacloud folks, please do respond here! ... I agree 100%. Although I understand the I think it's an unreasonably high burden in an area where there are many many other real pressing issues that need to be solved. I will go a little further. My focus is on workloads that are composed of scaling groups (one strict way of saying cattle not pets). In this case I do not need to migrate individual Compute instances, just shut down obsolete ones and start shiny new ones. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Networking Docs Swarm - Brisbane 9 August
Lana Brindley openst...@lanabrindley.com wrote on 08/04/2014 11:05:24 PM: I just wanted to let you all know about the OpenStack Networking Docs Swarm being held in Brisbane on 9 August. ... +++ on this. I can not contribute answers, but have lots of questions. Let me suggest that documentation is needed both for cloud providers doing general deployment and also for developers using DevStack. Not all of us developers are Neutron experts, so we need decent documentation. And developers sometimes need to use host machines with fewer than the ideal number of NICs. Sometimes those host machines are virtual, leading to nested virtualization (of network as well as compute). Thanks! Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] API confusion
http://developer.openstack.org/api-ref-networking-v2.html and http://docs.openstack.org/api/openstack-network/2.0/content/GET_listMembers__v2.0_pools__pool_id__members_lbaas_ext_ops_member.html say that to list LB pool members, the URL to GET is /v2.0/pools/{pool_id}/members When I use the CLI (`neutron -v lb-member-list`) I see a GET on /v2.0/lb/members.json What's going on here? Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [ceilometer][nova] extra Ceilometer Samples of the instance gauge
In a normal DevStack install, each Compute instance causes one Ceilometer Sample every 10 minutes. Except, there is an extra one every hour. And a lot of extra ones at the start. What's going on here? For example: $ ceilometer sample-list -m instance -q resource=9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab +--+--+---++--++ | Resource ID | Name | Type | Volume | Unit | Timestamp | +--+--+---++--++ | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T18:09:28| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T18:00:54.009877 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T17:59:28| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T17:49:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T17:39:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T17:29:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T17:19:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T17:09:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T17:00:07.002075 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T16:59:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T16:49:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T16:39:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T16:29:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T16:19:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T16:09:26| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T16:00:20.172520 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T15:59:27| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T15:49:26| ... | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T05:19:21| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T05:09:21| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T05:00:41.909634 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:59:21| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:49:21| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:39:21| | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:55.049799 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:54.834377 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:51.905095 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:40.962977 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:40.201907 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:40.091348 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:39.858939 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:39.693631 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:39.523561 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:39.421295 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:39.411968 | | 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| instance | 2014-07-30T04:32:38.916604 | +--+--+---++--++ Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] objects notifications
Gary Kotton gkot...@vmware.com wrote on 07/29/2014 12:43:08 PM: Hi, When reviewing https://review.openstack.org/#/c/107954/ it occurred to me that maybe we should consider having some kind of generic object wrapper that could do notifications for objects. Any thoughts on this? I am not sure what that would look like, but I agree that we have a problem with too many things not offering notifications. If there were some generic way to solve that problem, it would indeed be great. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Doug Wiegley do...@a10networks.com wrote on 07/23/2014 11:24:32 PM: Hi Mike, and listed the possible values of the status field, including INACTIVE. Other sources are telling me that status=INACTIVE when the health monitor thinks the member is unhealthy, status!=INACTIVE when the health monitor thinks the member is healthy. What's going on here? Indeed, the code will return a server status of INACTIVE if the lbaas agent marks a member ‘DOWN’. But, nowhere can I find that it actually ever does so. My statements about the status field for lbaas/neutron came from the author of the ref lbaas driver; I’ll check with him tomorrow and see if I misunderstood. I did an experiment, and found that the PoolMember status did indeed switch between ACTIVE and INACTIVE depending on the health of the member. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat] Convergence question - notifications?
The Convergence spec speaks of adding notification ( http://docs-draft.openstack.org/94/96394/3/gate/gate-heat-specs-docs/e3e9e06/doc/build/html/specs/convergence.html#proposed-change ). What sort of notifications are these? Are they something to which the implementation of a scaling group could subscribe, to learn when a member has gone missing? Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Doug Wiegley do...@a10networks.com wrote on 07/16/2014 04:58:52 PM: You do recall correctly, and there are currently no mechanisms for notifying anything outside of the load balancer backend when the health monitor/member state changes. But there *is* a mechanism for some outside thing to query the load balancer for the health of a pool member, right? I am thinking specifically of http://docs.openstack.org/api/openstack-network/2.0/content/GET_showMember__v2.0_pools__pool_id__members__member_id__lbaas_ext_ops_member.html --- whose response includes a status field for the member. Is there documentation for what values can appear in that field, and what each value means? Supposing we can leverage the pool member status, there remains an issue: establishing a link between an OS::Neutron::PoolMember and the corresponding scaling group member. We could conceivably expand the scaling group code so that if the member type is a stack then the contents of the stack are searched (perhaps recursively) for resources of type OS::Neutron::PoolMember, but that is a tad too automatic for my taste. It could pick up irrelevant PoolMembers. And such a level of implicit behavior is outside our normal style of doing things. We could follow the AWS style, by adding an optional property to the scaling group resource types --- where the value of that property can be the UUID of an OS::Neutron::LoadBalancer or an OS::Neutron::Pool. But that still does not link up an individual scaling group member with its corresponding PoolMember. Remember that if we are doing this at all, each scaling group member must be a stack. I think the simplest way to solve this would be to define a way that a such stack can put in its outputs the ID of the corresponding PoolMember. I would be willing to settle for simply saying that if such a stack has an output of type string and name __OS_pool_member then the value of that output is taken to be the ID of the corresponding PoolMember. Some people do not like reserved names; if that must be avoided then we can expand the schema language with a way to identify which stack output carries the PoolMember ID. Another alternative would be to add an optional scaling group property to carry the name of the stack output in question. There is also currently no way for an external system to inject health information about an LB or its members. I do not know that the injection has to be to the LB; in AWS the injection is to the scaling group. That would be acceptable to me too. Thoughts? Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Doug Wiegley do...@a10networks.com wrote on 07/23/2014 03:43:02 PM: From: Doug Wiegley do...@a10networks.com ... The state of the world today: ‘status’ in the neutron database is configuration/provisioning status, not operational status. Neutron- wide thing. We were discussing adding operational status fields (or a neutron REST call to get the info from the backend) last month, but it’s something that isn’t planned for a serious conversation until Kilo, at present. Thanks for the prompt response. Let me just grasp at one last straw: is there any chance that Neutron will soon define and implement Ceilometer metrics that reveal PoolMember health? Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Stephen Balukoff sbaluk...@bluebox.net wrote on 07/23/2014 09:14:35 PM: It's probably worth pointing out that most of the Neutron LBaaS team are spending most of our time doing a major revision to Neutron LBaaS. How stats processing should happen has definitely been discussed but not resolved at present-- and in any case it was apparent to those working on the project that it has secondary importance compared to the revision work presently underway. I personally would like to have queries about most objects in the stats API to Neutron LBaaS return a dictionary or I presume you meant of rather than or. statuses for child objects which then a UI or auto-scaling system can interpret however it wishes. That last part makes me a little nervious. I have seen can interpret however it wishes mean can not draw any useful inferences because there are no standards for that content. I presume that as the grand and glorious future arrives, it will be with due respect for backwards compatibility. In the present, I am getting what appears to be conflicting information on the status field of the responses of http://docs.openstack.org/api/openstack-network/2.0/content/GET_showMember__v2.0_pools__pool_id__members__member_id__lbaas_ext_ops_member.html Doug Wiegely wrote ‘status’ in the neutron database is configuration/provisioning status, not operational status and listed the possible values of the status field, including INACTIVE. Other sources are telling me that status=INACTIVE when the health monitor thinks the member is unhealthy, status!=INACTIVE when the health monitor thinks the member is healthy. What's going on here? Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Thomas Herve thomas.he...@enovance.com wrote on 07/17/2014 02:06:13 AM: There are 4 resources related to neutron load balancing. OS::Neutron::LoadBalancer is probably the least useful and the one you can *not* use, as it's only there for compatibility with AWS::AutoScaling::AutoScalingGroup. OS::Neutron::HealthMonitor does the health checking part, although maybe not in the way you want it. OK, let's work with these. My current view is this: supposing the Convergence work delivers monitoring of health according to a member's status in its service and reacts accordingly, the gaps (compared to AWS functionality) are the abilities to (1) get member health from application level pings (e.g., URL polling) and (2) accept member health declarations from an external system, with consistent reaction to health information from all sources. Source (1) is what an OS::Neutron::HealthMonitor specifies, and an OS::Neutron::Pool is the thing that takes such a spec. So we could complete the (1) part if there were a way to tell a scaling group to poll the member health information developed by an OS::Neutron::Pool. Does that look like the right approach? For (2), this would amount to having an API that an external system (with proper authorization) can use to declare member health. In the grand and glorious future when scaling groups have true APIs rather than being Heat hacks, such a thing would be part of those APIs. In the immediate future we could simply add this to the Heat API. Such an operation would take somethings like a stack name or UUID, the name or UUID of a resource that is a scaling group, and the member name or UUID of the Resource whose health is being declared, and health_status=unhealthy. Does that look about right? For both of these new sources, the remaining question is how to get the right reaction. In the case that the member is actually deleted already, life is easy. Let's talk about the other cases. Note that AWS admits that there might be false detection of unhealth as a member's contents finish getting into regular operation; AWS handles this by saying that the right reaction is to react only after unhealth has been consistently detected for a configured amount of time. The simplest thing for a scaling group to do might be to include that hysteresis and eventually effect removal of a member by generating a new template that excludes the to-be-deleted member and doing an UPDATE on itself (qua stack) with that new template. Does that look about right? Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Clint Byrum cl...@fewbar.com wrote on 07/18/2014 12:56:32 PM: Excerpts from Mike Spreitzer's message of 2014-07-18 09:12:21 -0700: ... OK, let's work with these. My current view is this: supposing the Convergence work delivers monitoring of health according to a member's status in its service and reacts accordingly, the gaps (compared to AWS functionality) are the abilities to (1) get member health from application level pings (e.g., URL polling) and (2) accept member health declarations from an external system, with consistent reaction to health information from all sources. Convergence will not deliver monitoring, though I understand how one might have that misunderstanding. Convergence will check with the API that controls a physical resource to determine what Heat should consider its status to be for the purpose of ongoing orchestration. If I understand correctly, your point is that healing is not automatic. Since a scaling group is a nested stack, the observing part of Convergence will automatically note in the DB when the physical resource behind a scaling group member (in its role as a stack resource) is deleted. And when convergence engine gets around to acting on that Resource, the backing physical resource will be automatically re-created. But there is nothing that automatically links the notice of divergence to the converging action. Have I got that right? Source (1) is what an OS::Neutron::HealthMonitor specifies, and an OS::Neutron::Pool is the thing that takes such a spec. So we could complete the (1) part if there were a way to tell a scaling group to poll the member health information developed by an OS::Neutron::Pool. Does that look like the right approach? For (2), this would amount to having an API that an external system (with proper authorization) can use to declare member health. In the grand and glorious future when scaling groups have true APIs rather than being Heat hacks, such a thing would be part of those APIs. In the immediate future we could simply add this to the Heat API. Such an operation would take somethings like a stack name or UUID, the name or UUID of a resource that is a scaling group, and the member name or UUID of the Resource whose health is being declared, and health_status=unhealthy. Does that look about right? Isn't (2) covered already by the cloudwatch API in Heat? I am going to claim ignorance of it a bit, as I've never used it, but it seems like the same thing. I presume that by cloudwatch API you mean Ceilometer. Today a Ceilometer alarm can be given a URL to invoke but can not be told about any special headers or body to use in the invocation (i.e., no parameters for the HTTP operation). More to the point, the idea here is supporting a general external system that might determine health in its own way, not necessarily through programming Ceilometer to detect it. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Clint Byrum cl...@fewbar.com wrote on 07/02/2014 01:54:49 PM: Excerpts from Qiming Teng's message of 2014-07-02 00:02:14 -0700: Just some random thoughts below ... On Tue, Jul 01, 2014 at 03:47:03PM -0400, Mike Spreitzer wrote: In AWS, an autoscaling group includes health maintenance functionality --- both an ability to detect basic forms of failures and an abilityto react properly to failures detected by itself or by a load balancer. What is the thinking about how to get this functionality in OpenStack? Since We are prototyping a solution to this problem at IBM Research - China lab. The idea is to leverage oslo.messaging and ceilometer events for instance (possibly other resource such as port, securitygroup ...) failure detection and handling. Hm.. perhaps you should be contributing some reviews here as you may have some real insight: https://review.openstack.org/#/c/100012/ This sounds a lot like what we're working on for continuous convergence. I noticed that health checking in AWS goes beyond convergence. In AWS an ELB can be configured with a URL to ping, for application-level health checking. And an ASG can simply be *told* the health of a member by a user's own external health system. I think we should have analogous functionality in OpenStack. Does that make sense to you? If so, do you have any opinion on the right way to integrate, so that we do not have three completely independent health maintenance systems? Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Doug Wiegley do...@a10networks.com wrote on 07/16/2014 04:58:52 PM: On 7/16/14, 2:43 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Mike Spreitzer's message of 2014-07-16 10:50:42 -0700: ... I noticed that health checking in AWS goes beyond convergence. In AWS an ELB can be configured with a URL to ping, for application-level health checking. And an ASG can simply be *told* the health of a member by a user's own external health system. I think we should have analogous functionality in OpenStack. Does that make sense to you? If so, do you have any opinion on the right way to integrate, so that we do not have three completely independent health maintenance systems? The check url is already a part of Neutron LBaaS IIRC. Yep. LBaaS is a work in progress, right? Those of use using Nova networking are not feeling the love, unfortunately. As far as Heat goes, there is no LBaaS resource type. The OS::Neutron::LoadBalancer resource type does not have any health checking properties. The AWS::ElasticLoadBalancing::LoadBalancer does have a parameter that prescribes health checking --- but, as far as I know, there is no way to ask such a load balancer for its opinion of a member's health. What may not be a part is notifications for when all members are reporting down (which might be something to trigger scale-up). I do not think we want an ASG to react only when all members are down; I think an ASG should maintain at least its minimum size (although I have to admit that I do not understand why the current code has an explicit exception to that). You do recall correctly, and there are currently no mechanisms for notifying anything outside of the load balancer backend when the health monitor/member state changes. This is true in AWS as well. The AWS design is that you can configure the ASG to poll the ELB for its opinion of member health. The idea seems to be that an ASG can get health information from three kinds of sources (polling status in EC2, polling ELB, and being explicitly informed), synthesizes its own summary opinion, and reacts in due time. There is also currently no way for an external system to inject health information about an LB or its members. Both would be interesting additions. doug If we don't have push checks in our auto scaling implementation then we don't have a proper auto scaling implementation. I am not sure what is meant by push checks. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] autoscaling across regions and availability zones
Zane Bitter zbit...@redhat.com wrote on 07/10/2014 05:57:14 PM: On 09/07/14 22:38, Mike Spreitzer wrote: Zane Bitter zbit...@redhat.com wrote on 07/01/2014 06:54:58 PM: On 01/07/14 16:23, Mike Spreitzer wrote: ... Hmm, now that I think about it, CloudFormation provides a Fn::GetAZs function that returns a list of available AZs. That suggests an implementation where you can specify an AZ If we're whittling down then it would be one or more AZs, right? when creating the stack and the function returns only that value within that stack (and its children). There's no way in OS::Heat::AutoScalingGroup to specify an intrinsic function that is resolved in the context of the scaling group's nested stack, I am not sure I understand what you mean. Is it: there is no way for the implementation of a resource type to create or modify an intrinsic function? but if the default value of the AZ for OS::Nova::Server were calculated the same way then the user would have the option of omitting the AZ (to allow the autoscaling implementation to control it) I am not sure I get this part. If the scaling group member type is a Compute instance (as well as if it is not) then the template generated by the group (to implement the group) wants to put different resources in different AZs. The nested stack that is the scaling group is given a whole list of AZs as its list-of-AZs parameter value. or overriding it explicitly. At that point you don't even need the intrinsic function. So don't assign a stack to a particular AZ as such, but allow the list of valid AZs to be whittled down as you move toward the leaves of the tree of templates. I partially get the suggestion. Let me repeat it back to see if it sounds right. Let the stack create and update operations gain an optional parameter that is a list of AZs, (noting that a stack operation parameter is something different from a parameter specified in a template) constrained to be a subset of the AZs available to the user in Heat's configured region; the default value is the list of all AZs available to the user in Heat's configured region. Redefine the Fn::GetAZs intrinsic to return that new parameter's value. For each resource type that can be given a list of AZs, we (as plugin authors) redefine the default to be the list returned by Fn::GetAZs; for each resource type that can be given a single AZ, we (as plugin authors) redefine the default to be one (which one?) of the AZs returned by Fn::GetAZs. That would probably require some finessing around the schema technology, because a parameter's default value is fixed when the resource type is registered, right? A template generated by scaling group code somehow uses that new stack operation parameter to set the member's AZ when the member is a stack and the scaling group is spanning AZs. Would the new stack operation parameter (list of AZs) be reflected as a property of OS::Heat::Stack? How would that list be passed in a scenario like https://review.openstack.org/#/c/97366/10/hot/asg_of_stacks.yaml,unified where the member type is a template filename and the member's properties are simply the stack's parameters? Can the redefinitions mentioned here be a backward compatibility problem? So yes, the tricky part is how to handle that when the scaling unit is not a server (or a provider template with the same interface as a server). One solution would have been to require that the scaled unit was, indeed, either an OS::Nova::Server or a provider template with the same interface as (or a superset of) an OS::Nova::Server, but the consensus was against that. (Another odd consequence of this decision is that we'll potentially be overwriting an AZ specified in the launch config section with one from the list supplied to the scaling group itself.) For provider templates, we could insert a pseudo-parameter containing the availability zone. I think that could be marginally better than taking over one of the user's parameters, but you're basically on the right track IMO. I considered a built-in function or pseudo parameter and rejected them based on a design principle that was articulated in an earlier discussion: no modes. Making the innermost template explicitly declare that it takes an AZ parameter makes it more explicit what is going on. But I agree that this is a relatively minor design point, and would be content to go with a pseudo-parameter if the community really prefers that. Unfortunately, that is not the end of the story, because we still have to deal with other types of resources being scaled. I always advocated for an autoscaling resource where the scaled unit was either a provider stack (if you provided a template) or an OS::Nova::Server (if you didn't
Re: [openstack-dev] 答复: [heat] autoscaling across regions and availability zones
Huangtianhua huangtian...@huawei.com wrote on 07/04/2014 02:35:56 AM: I have register a bp about this : https://blueprints.launchpad.net/ heat/+spec/implement-autoscalinggroup-availabilityzones ・ ・ And I am thinking how to implement this recently. ・ ・ According to AWS autoscaling implementation “attempts to distribute instances evenly between the Availability Zones that are enabled for your Auto Scaling group. ・ Auto Scaling does this by attempting to launch new instances in the Availability Zone with the fewest instances. If the attempt fails, however, Auto Scaling will attempt to launch in other zones until it succeeds.” But there is a doubt about the “fewest instance”, .e.g There are two azs, Az1: has two instances Az2: has three instances ・And then to create a asg with 4 instances, I think we should create two instances respectively in az1 and az2, right? Now if need to extend to 5 instances for the asg, which az to lauch new instance? If you interested in this bp, I think we can discuss thisJ The way AWS handles this is described in http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AS_Concepts.html#arch-AutoScalingMultiAZ That document leaves a lot of freedom to the cloud provider. And rightfully so, IMO. To answer your specific example, when spreading 5 instances across 2 zones, the cloud provider gets to pick which zone gets 3 and which zone gets 2. As for what a Heat scaling group should do, that depends on what Nova can do for Heat. I have been told that Nova's instance-creation operation takes an optional parameter that identifies one AZ and, if that parameter is not provided, then a configured default AZ is used. Effectively, the client has to make the choice. I would start out with Heat making a random choice; in subsequent development it might query or monitor Nova for some statistics to guide the choice. An even more interesting issue is the question of choosing which member(s) to remove when scaling down. The approach taken by AWS is documented at http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/us-termination-policy.html but the design there has redundant complexity and the doc is not well written. Following is a short sharp presentation of an isomorphic system. A client that owns an ASG configures that ASG to have a series (possibly empty) of instance termination policies; the client can change the series during the ASG's lifetime. Each policy is drawn from the following menu: OldestLaunchConfiguration ClosestToNextInstanceHour OldestInstance NewestInstance (see the AWS doc for the exact meaning of each). The signature of a policy is this: given a set of candidate instances for removal, return a subset (possibly the whole input set). When it is time to remove instances from an ASG, they are chosen one by one. AWS uses the following procedure to choose one instance to remove. 1. Choose the AZ from which the instance will be removed. The choice is based primarily on balancing the number of group members in each AZ, and ties are broken randomly. 2. Starting with a candidate set consisting of all the ASG's members in the chosen AZ, run the configured series of policies to progressively narrow down the set of candidates. 3. Use OldestLaunchConfiguration and then ClosestToNextInstanceHour to further narrow the set of candidates. 4. Make a random choice among the final set of candidates. Since each policy returns its input when its input's size is 1 we do not need to talk about early exits when defining the procedure (although the implementation might make such optimizations). I plan to draft a spec. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] autoscaling across regions and availability zones
Zane Bitter zbit...@redhat.com wrote on 07/01/2014 06:54:58 PM: On 01/07/14 16:23, Mike Spreitzer wrote: An AWS autoscaling group can span multiple availability zones in one region. What is the thinking about how to get analogous functionality in OpenStack? ... Currently, a stack does not have an AZ. That makes the case of an OS::Heat::AutoScalingGroup whose members are nested stacks interesting --- how does one of those nested stacks get into the right AZ? And what does that mean, anyway? The meaning would have to be left up to the template author. But he needs something he can write in his member template to reference the desired AZ for the member stack. I suppose we could stipulate that if the member template has a parameter named availability_zone and typed string then the scaling group takes care of providing the right value to that parameter. The concept of an availability zone for a stack is not meaningful. Servers have availability zones; stacks exist in one region. It is up to the *operator*, not the user, to deploy Heat in such a way that it remains highly-available assuming the Region is still up. There are two distinct issues there: (1) making the heat engine HA and (2) making a scaling group of stacks span across AZs (within a region). I agree that (1) is the cloud provider's problem, and never meant to suggest otherwise. I think (2) makes sense by analogy: a nested stack is a way of implementing a particular abstraction (defined by the template) --- in fact the outer template author might not even be aware that the group members are stacks, thanks to provider templates --- and here we suppose the user has chosen to use an abstraction that makes sense to be considered to be in an AZ. While a stack in general does not have an AZ, I think we can suppose that if the outer template author asked for stacks to be spread across AZs then the stacks in question can reasonably considered to each be in one AZ. For example, the inner template might contain a Compute instance and a Cinder volume and an attachment between the two; such a stack makes sense to put in an AZ. Heat itself does not even need there to be any particular real meaning to a stack being in an AZ, all I am proposing is that Heat make this concept available to the authors of the outer and innermost templates to use in whatever way they find useful. So yes, the tricky part is how to handle that when the scaling unit is not a server (or a provider template with the same interface as a server). One solution would have been to require that the scaled unit was, indeed, either an OS::Nova::Server or a provider template with the same interface as (or a superset of) an OS::Nova::Server, but the consensus was against that. (Another odd consequence of this decision is that we'll potentially be overwriting an AZ specified in the launch config section with one from the list supplied to the scaling group itself.) For provider templates, we could insert a pseudo-parameter containing the availability zone. I think that could be marginally better than taking over one of the user's parameters, but you're basically on the right track IMO. I considered a built-in function or pseudo parameter and rejected them based on a design principle that was articulated in an earlier discussion: no modes. Making the innermost template explicitly declare that it takes an AZ parameter makes it more explicit what is going on. But I agree that this is a relatively minor design point, and would be content to go with a pseudo-parameter if the community really prefers that. Unfortunately, that is not the end of the story, because we still have to deal with other types of resources being scaled. I always advocated for an autoscaling resource where the scaled unit was either a provider stack (if you provided a template) or an OS::Nova::Server (if you didn't), but the implementation that landed followed the design of ResourceGroup by allowing (actually, requiring) you to specify an arbitrary resource type. We could do something fancy here involving tagging the properties schema with metadata so that we could allow plugin authors to map the AZ list to an arbitrary property. However, I propose that we just raise a validation error if the AZ is specified for a resource that is not either an OS::Nova::Server or a provider template. Yes, I agree with limiting ambition. For OS::Heat::AutoScalingGroup and ResourceGroup, I think a pretty simple rule will cover all the cases that matter besides nested stack: if the member resource type has a string parameter named availability zone (camel or snake case) then this is valid. I just reviewed LaunchConfiguration ( http://docs.openstack.org/developer/heat/template_guide/cfn.html#AWS::AutoScaling::LaunchConfiguration ) and noticed that it does not have an availability_zone. Since AWS::AutoScalingGroup
Re: [openstack-dev] 答复: 答复: [heat] autoscaling across regions and availability zones
Huangtianhua huangtian...@huawei.com wrote on 07/09/2014 10:20:42 PM: 发件人: Mike Spreitzer [mailto:mspre...@us.ibm.com] 发送时间: 2014年7月10日 3:19 收件人: OpenStack Development Mailing List (not for usage questions) 主题: Re: [openstack-dev] 答复: [heat] autoscaling across regions and availability zones Huangtianhua huangtian...@huawei.com wrote on 07/04/2014 02:35:56 AM: I have register a bp about this : https://blueprints.launchpad.net/ heat/+spec/implement-autoscalinggroup-availabilityzones ・ ・ And I am thinking how to implement this recently. ・ ・ According to AWS autoscaling implementation “attempts to distribute instances evenly between the Availability Zones that are enabled for your Auto Scaling group. ・ Auto Scaling does this by attempting to launch new instances in the Availability Zone with the fewest instances. If the attempt fails, however, Auto Scaling will attempt to launch in other zones until it succeeds.” But there is a doubt about the “fewest instance”, .e.g There are two azs, Az1: has two instances Az2: has three instances ・And then to create a asg with 4 instances, I think we should create two instances respectively in az1 and az2, right? Now if need to extend to 5 instances for the asg, which az to lauch new instance? If you interested in this bp, I think we can discuss thisJ The way AWS handles this is described in http://docs.aws.amazon.com/ AutoScaling/latest/DeveloperGuide/AS_Concepts.html#arch-AutoScalingMultiAZ That document leaves a lot of freedom to the cloud provider. And rightfully so, IMO. To answer your specific example, when spreading 5 instances across 2 zones, the cloud provider gets to pick which zone gets 3 and which zone gets 2. As for what a Heat scaling group should do, that depends on what Nova can do for Heat. I have been told that Nova's instance-creation operation takes an optional parameter that identifies one AZ and, if that parameter is not provided, then a configured default AZ is used. Effectively, the client has to make the choice. I would start out with Heat making a random choice; in subsequent development it might query or monitor Nova for some statistics to guide the choice. yes, I read the doc, as you said, the doc is not well written, so I doubt about the “fewest instance” before, but now IMO, “fewest instance” means the instances of the group, so you are right, to my specific example, the instance should be launch at random or in a round-robin mode. An even more interesting issue is the question of choosing which member(s) to remove when scaling down. The approach taken by AWS is documented at http://docs.aws.amazon.com/AutoScaling/latest/ DeveloperGuide/us-termination-policy.html but the design there has redundant complexity and the doc is not well written. Following is a short sharp presentation of an isomorphic system. A client that owns an ASG configures that ASG to have a series (possibly empty) of instance termination policies; the client can change the series during the ASG's lifetime. Each policy is drawn from the following menu: OldestLaunchConfiguration ClosestToNextInstanceHour OldestInstance NewestInstance and there is a default policy of termination... If you are objecting to the fact that I omitted Default from the menu of available policies, note that there is no need to discuss a Default policy. Remember that doc also stipulates that if your list of policies includes Default then Default should be at the end of your list. Since the Default policy is always applied after your list (if there is a need), there is no need to explicitly declare it at the end. Note also that the doc is confusing about default policy; at first it says that the default policy selects an AZ, and later it mentions using the default policy to narrow down a set of candidates within an AZ that has already been chosen. So a simpler, equivalent explanation has no need to ever talk about a default policy, rather it can stipulate that the configured list of ways to narrow down the set of candidate instances is always followed by OldestLaunchConfiguration, then ClosestToNextInstanceHour, then a random choice (which is how the so-called default policy narrows down the set of candidates after the AZ has been chosen). I brought this whole sub-topic up only to mostly dismiss it. OpenStack scaling groups do not currently attempt anything similar to OldestLaunchConfiguration or ClosestToNextInstanceHour and I recommend that the work on spanning across AZs should not attempt to change that. My recommendation is that this spec only call for adding a balancing selection of AZ to what is currently done. I have also opened a separate discussion (ML subject One more lifecycle plug point - in scaling groups ) about opening up this policy issue to give
Re: [openstack-dev] [heat] One more lifecycle plug point - in scaling groups
Steven Hardy sha...@redhat.com wrote on 07/02/2014 06:16:21 AM: On Wed, Jul 02, 2014 at 02:41:19AM +, Adrian Otto wrote: Zane, If you happen to have a link to this blueprint, could you replywith it? ... I believe Zane was referring to: https://blueprints.launchpad.net/heat/+spec/update-hooks This is also related to the action aware software config spec: https://review.openstack.org/#/c/98742/ So in future, you might autoscale nested stacks containing action-aware software config resources, then you could define specific actions which happen e.g on scale-down (on action DELETE). Thanks, those are great pointers. The second pretty much covers the first, right? I do think the issue these address --- the need to get application logic involved in, e.g., shutdown --- is most of what an application needs; involvement in selection of which member(s) to delete is much less important (provided that clean shutdown mechanism prevents concurrent shutdowns). So that provides a pretty good decoupling between the application's concerns and a holistic scheduler's concerns. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Zane Bitter zbit...@redhat.com wrote on 07/01/2014 06:58:47 PM: On 01/07/14 15:47, Mike Spreitzer wrote: In AWS, an autoscaling group includes health maintenance functionality --- both an ability to detect basic forms of failures and an ability to react properly to failures detected by itself or by a load balancer. What is the thinking about how to get this functionality in OpenStack? Since OpenStack's OS::Heat::AutoScalingGroup has a more general member type, what is the thinking about what failure detection means (and how it would be accomplished, communicated)? I have not found design discussion of this; have I missed something? Yes :) https://review.openstack.org/#/c/95907/ The idea is that Convergence will provide health maintenance for _all_ forms of resources in Heat. Once this is implemented, autoscaling gets it for free by virtue of that fact that it manages resources using Heat stacks. Ah, right. My reading of that design is not quite so simple. Note that in the User Stories section it calls for different treatment of Compute instances depending on whether they are in a scaling group. That's why I was thinking of this from a scaling group perspective. But perhaps the more natural approach is to take the pervasive perspective and figure out how to suppress convergence for the Compute instances to which it should not apply. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Mike Spreitzer/Watson/IBM@IBMUS wrote on 07/02/2014 02:41:48 AM: Zane Bitter zbit...@redhat.com wrote on 07/01/2014 06:58:47 PM: On 01/07/14 15:47, Mike Spreitzer wrote: In AWS, an autoscaling group includes health maintenance functionality --- both an ability to detect basic forms of failures and an ability to react properly to failures detected by itself or by a load balancer. What is the thinking about how to get this functionality in OpenStack? Since OpenStack's OS::Heat::AutoScalingGroup has a more general member type, what is the thinking about what failure detection means (and how it would be accomplished, communicated)? I have not found design discussion of this; have I missed something? Yes :) https://review.openstack.org/#/c/95907/ The idea is that Convergence will provide health maintenance for _all_ forms of resources in Heat. Once this is implemented, autoscaling gets it for free by virtue of that fact that it manages resources using Heat stacks. Ah, right. My reading of that design is not quite so simple... There are a couple more issues that arise in this approach. The biggest one is how to integrate application-level failure detection. I added a comment to this effect on the Convergence spec. Another issue is that, at least initially, Convergence is not always on; rather, it is an operation that can be invoked on a stack. When would a scaling group invoke this action on itself (more precisely, itself considered as a nested stack)? One obvious possibility is on a periodic basis. It the convergence operation is pretty cheap when no divergence has been detected, then that might be acceptable. Otherwise we might want the scaling group to set up some sort of notification, but that would be another batch of member-type specific code. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Qiming Teng teng...@linux.vnet.ibm.com wrote on 07/02/2014 03:02:14 AM: Just some random thoughts below ... On Tue, Jul 01, 2014 at 03:47:03PM -0400, Mike Spreitzer wrote: ... I have not found design discussion of this; have I missed something? I suppose the natural answer for OpenStack would be centered around webhooks... Well, I would suggest we generalize this into a event messaging or signaling solution, instead of just 'webhooks'. The reason is that webhooks as it is implemented today is not carrying a payload of useful information -- I'm referring to the alarms in Ceilometer. OK, this is great (and Steve Hardy provided more details in his reply), I did not know about the existing abilities to have a payload. However Ceilometer alarms are still deficient in that way, right? A Ceilometer alarm's action list is simply a list of URLs, right? I would be happy to say let's generalize Ceilometer alarms to allow a payload in an action. There are other cases as well. A member failure could be caused by a temporary communication problem, which means it may show up quickly when a replacement member is already being created. It may mean that we have to respond to an 'online' event in addition to an 'offline' event? ... The problem here today is about the recovery of SG member. If it is a compute instance, we can 'reboot', 'rebuild', 'evacuate', 'migrate' it, just to name a few options. The most brutal way to do this is like what HARestarter is doing today -- delete followed by a create. We could get into arbitrary subtlety, and maybe eventually will do better, but I think we can start with a simple solution that is widely applicable. The simple solution is that once the decision has been made to do convergence on a member (note that this is distinct from merely detecting and noting a divergence) then it will be done regardless of whether the doomed member later appears to have recovered, and the convergence action for a scaling group member is to delete the old member and create a replacement (not in that order). When the member is a nested stack and Ceilometer exists, it could be the member stack's responsibility to include a Ceilometer alarm that detects the member stack's death and hit the member stack's deletion webhook. This is difficult. A '(nested) stack' is a Heat specific abstraction -- recall that we have to annotate a nova server resource in its metadata to which stack this server belongs. Besides the 'visible' resources specified in a template, Heat may create internal data structures and/or resources (e.g. users) for a stack. I am not quite sure a stack's death can be easily detected from outside Heat. It would be at least cumbersome to have Heat notify Ceilometer that a stack is dead, and then have Ceilometer send back a signal. A (nested) stack is not only a heat-specific abstraction but its semantics and failure modes are specific to the stack (at least, its template). I think we have no practical choice but to let the template author declare how failure is detected. It could be as simple as creating a Ceilometer alarms that detect death one or more resources in the nested stack; it could be more complicated Ceilometer stuff; it could be based on something other than, or in addition to, Ceilometer. If today there are not enough sensors to detect failures of all kinds of resources, I consider that a gap in telemetry (and think it is small enough that we can proceed usefully today, and should plan on filling that gap over time). There is a small matter of how the author of the template used to create the member stack writes some template snippet that creates a Ceilometer alarm that is specific to a member stack that does not exist yet. How about just one signal responder per ScalingGroup? A SG is supposed to be in a better position to make the judgement: do I have to recreate a failed member? am I recreating it right now or wait a few seconds? maybe I should recreate the member on some specific AZs? That is confusing two issues. The thing that is new here is making the scaling group recognize member failure; the primary reaction is to update its accounting of members (which, in the current code, must be done by making sure the failed member is deleted); recovery of other scaling group aspects is fairly old-hat, it is analogous to the problems that the scaling group already solves when asked to increase its size. ... I suppose we could stipulate that if the member template includes a parameter with name member_name and type string then the OS OG takes care of supplying the correct value of that parameter; as illustrated in the asg_of_stacks.yaml of https://review.openstack.org/#/c/97366/ , a member template can use a template parameter to tag Ceilometer data for querying. The URL of the member stack's deletion webhook could be passed to the member
Re: [openstack-dev] [heat] health maintenance in autoscaling groups
Steven Hardy sha...@redhat.com wrote on 07/02/2014 06:02:36 AM: On Wed, Jul 02, 2014 at 03:02:14PM +0800, Qiming Teng wrote: Just some random thoughts below ... On Tue, Jul 01, 2014 at 03:47:03PM -0400, Mike Spreitzer wrote: ... The resource signal interface used by ceilometer can carry whatever data you like, so the existing solution works fine, we don't need a new one IMO. That's great, I did not know about it. Thanks for the details on this, I do think it is an improvement. Yes, it is a slight security downgrade --- the signed URL is effectively a capability, and allowing a payload increases the cross section of what an attacker can do with one stolen capability. But I am willing to live with it. ... Are you aware of the Existence of instance meter in ceilometer? I am, and was thinking it might be very directly usable. Has anybody tested or demonstrated this? ... How about just one signal responder per ScalingGroup? A SG is supposed to be in a better position to make the judgement: do I have to recreate a failed member? am I recreating it right now or wait a few seconds? maybe I should recreate the member on some specific AZs? This is what we have already - you have one ScalingPolicy (which is a SignalResponder), and the ScalingPolicy is the place where you make the decision about what to do with the data provided from the alarm. I think the existing ScalingPolicy is about a different issue. ScalingPolicy is mis-named; it is really only a ScalingAction, and it is about how to adjust the desired size. It does not address the key missing piece here, which is how the scaling group updates its accounting of the number of members it has. That accounting is done simply by counting members. So if a member becomes dysfunctional but remains extant, the scaling group logic continues to count it. Hmm, can a scaling group today properly cope with member deletion if prodded to do a ScalingPolicy(Action) that is 'add 1 member'? (I had considered 'add 0 members' but that fails to produce a change in an important case --- when the size is now below minimum (fun fact about the code!). ) ... I am not in favor of the per-member webhook design. But I vote for an additional *implicit* parameter to a nested stack of any groups. It could be an index or a name. I agree, we just need appropriate metadata in ceilometer, which can then be passed back to heat via the resource signal when the alarm happens. We need to get the relevant meter samples in Ceilometer tagged with something that is unique to the [scaling group, member] and referencable in the template source. For the case of a scaling group whose member type is nested stack, you could invent a way to implicitly pass such tagging down through all the intervening abstractions. I was supposing the preferred solution would be for the template author to explicitly do this. ... Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat] health maintenance in autoscaling groups
In AWS, an autoscaling group includes health maintenance functionality --- both an ability to detect basic forms of failures and an ability to react properly to failures detected by itself or by a load balancer. What is the thinking about how to get this functionality in OpenStack? Since OpenStack's OS::Heat::AutoScalingGroup has a more general member type, what is the thinking about what failure detection means (and how it would be accomplished, communicated)? I have not found design discussion of this; have I missed something? I suppose the natural answer for OpenStack would be centered around webhooks. An OpenStack scaling group (OS SG = OS::Heat::AutoScalingGroup or AWS::AutoScaling::AutoScalingGroup or OS::Heat::ResourceGroup or OS::Heat::InstanceGroup) could generate a webhook per member, with the meaning of the webhook being that the member has been detected as dead and should be deleted and removed from the group --- and a replacement member created if needed to respect the group's minimum size. When the member is a Compute instance and Ceilometer exists, the OS SG could define a Ceilometer alarm for each member (by including these alarms in the template generated for the nested stack that is the SG), programmed to hit the member's deletion webhook when death is detected (I imagine there are a few ways to write a Ceilometer condition that detects instance death). When the member is a nested stack and Ceilometer exists, it could be the member stack's responsibility to include a Ceilometer alarm that detects the member stack's death and hit the member stack's deletion webhook. There is a small matter of how the author of the template used to create the member stack writes some template snippet that creates a Ceilometer alarm that is specific to a member stack that does not exist yet. I suppose we could stipulate that if the member template includes a parameter with name member_name and type string then the OS OG takes care of supplying the correct value of that parameter; as illustrated in the asg_of_stacks.yaml of https://review.openstack.org/#/c/97366/ , a member template can use a template parameter to tag Ceilometer data for querying. The URL of the member stack's deletion webhook could be passed to the member template via the same sort of convention. When Ceilometer does not exist, it is less obvious to me what could usefully be done. Are there any useful SG member types besides Compute instances and nested stacks? Note that a nested stack could also pass its member deletion webhook to a load balancer (that is willing to accept such a thing, of course), so we get a lot of unity of mechanism between the case of detection by infrastructure vs. application level detection. I am not entirely happy with the idea of a webhook per member. If I understand correctly, generating webhooks is a somewhat expensive and problematic process. What would be the alternative? Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat] autoscaling across regions and availability zones
An AWS autoscaling group can span multiple availability zones in one region. What is the thinking about how to get analogous functionality in OpenStack? Warmup question: what is the thinking about how to get the levels of isolation seen between AWS regions when using OpenStack? What is the thinking about how to get the level of isolation seen between AWS AZs in the same AWS Region when using OpenStack? Do we use OpenStack Region and AZ, respectively? Do we believe that OpenStack AZs can really be as independent as we want them (note that this is phrased to not assume we only want as much isolation as AWS provides --- they have had high profile outages due to lack of isolation between AZs in a region)? I am going to assume that the answer to the question about ASG spanning involves spanning OpenStack regions and/or AZs. In the case of spanning AZs, Heat has already got one critical piece: the OS::Heat::InstanceGroup and AWS::AutoScaling::AutoScalingGroup types of resources take a list of AZs as an optional parameter. Presumably all four kinds of scaling group (i.e., also OS::Heat::AutoScalingGroup and OS::Heat::ResourceGroup) should have such a parameter. We would need to change the code that generates the template for the nested stack that is the group, so that it spreads the members across the AZs in a way that is as balanced as is possible at the time. Currently, a stack does not have an AZ. That makes the case of an OS::Heat::AutoScalingGroup whose members are nested stacks interesting --- how does one of those nested stacks get into the right AZ? And what does that mean, anyway? The meaning would have to be left up to the template author. But he needs something he can write in his member template to reference the desired AZ for the member stack. I suppose we could stipulate that if the member template has a parameter named availability_zone and typed string then the scaling group takes care of providing the right value to that parameter. To spread across regions adds two things. First, all four kinds of scaling group would need the option to be given a list of regions instead of a list of AZs. More likely, a list of contexts as defined in https://review.openstack.org/#/c/53313/ --- that would make this handle multi-cloud as well as multi-region. The other thing this adds is a concern for context health. It is not enough to ask Ceilometer to monitor member health --- in multi-region or multi-cloud you also have to worry about the possibility that Ceilometer itself goes away. It would have to be the scaling group's responsibility to monitor for context health, and react properly to failure of a whole context. Does this sound about right? If so, I could draft a spec. Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat] One more lifecycle plug point - in scaling groups
Thinking about my favorite use case for lifecycle plug points for cloud providers (i.e., giving something a chance to make a holistic placement decision), it occurs to me that one more is needed: a scale-down plug point. A plugin for this point has a distinctive job: to decide which group member(s) to remove from a scaling group (i.e., OS::Heat::AutoScalingGroup or OS::Heat::InstanceGroup or OS::Heat::ResourceGroup or AWS::AutoScaling::AutoScalingGroup). The plugin's signature could be something like this: given a list of group members and a number to remove, return the list of members to remove (or, equivalently, return the list of members to keep). What do you think? Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev