[openstack-dev] [Designate] In what sense is it multi-tenant?

2017-02-10 Thread Mike Spreitzer
In what sense is Designate multi-tenant?  Can it be programmed to give 
different views to different DNS clients?  (If so, how?)

Thanks,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [designate] Status of the project

2017-02-10 Thread Mike Spreitzer
Joshua Harlow  wrote on 02/10/2017 12:21:08 PM:

> Knowing where this is at and the issues. It makes me wonder if it is 
> worthwhile to start thinking about how we can start to look at 'outside 
> the openstack' projects for DNS. I believe there is a few that are 
> similar enough to designate (though I don't know well enough) for 
> example things like SkyDNS (or others which I believe there are a few).
> 
> Perhaps we need to start thinking outside the openstack 'box' in regards 

> to NIH syndrome and accept the fact that we as a community may not be 
> able to recreate the world successfully in all cases (the same could be 
> said about things like k8s and others).
> 
> If we got out of the mindset of openstack as a thing must have tightly 
> integrated components (over all else) and started thinking about how we 
> can be much more loosely coupled (and even say integrating non-python, 
> non-openstack projects) would that be beneficial (I think it would)?

I think you might be on to something.  The Kubernetes community seems to 
be thinking about an external DNS service too.  I see 
https://github.com/kubernetes-incubator/external-dns was just created, but 
do not know anything more about it.

Regards,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Kuryr] [Neutron] Controlling security groups through Kuryr's libnetwork plugin [was: Waiting until Neutron PortisActive]

2016-06-12 Thread Mike Spreitzer
Antoni Segura Puimedon  wrote on 06/11/2016 
07:39:41 PM:

> Well, with a label you can make the Neutron Port have an SG that 
> forbids pinging.

Wait, what?  Labels on what can do what?

Thanks,
Mike


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron PortisActive

2016-06-11 Thread Mike Spreitzer
What about pinging?  BTW, from where would the pings come?

In the Docker/Swarm API today there is no way to disable ping.  However, 
once Kuryr's libnetwork plugin is updated so that `docker network connect 
--ip=W.X.Y.Z ...` will latch onto a matching pre-existing Neutron Port, if 
it exists, there will be a way for a user to disable pings (right?).

In the Kubernetes API there is now a way to do something like security 
groups, it is called NetworkPolicy; it is not yet well defined enough to 
say whether it gives the user a way to disable pings.

Thanks,
Mike



From:   Mohammad Banikazemi/Watson/IBM@IBMUS
To: "OpenStack Development Mailing List \(not for usage questions\)" 

Date:   06/10/2016 10:50 AM
Subject:Re: [openstack-dev] [Kuryr] [Neutron] Waiting until 
Neutron PortisActive



Hi Neil,

Currently, when a docker libnetwork "join" operation in Kuryr is returned, 
it is not guaranteed that the network connectivity has been established. 
There are containers that check for network connectivity as the first 
thing they do when they come up and under heavy load some notice there is 
no connectivity and simply bail out. I am trying to deal with such a use 
case,

Thanks for pointing out that option 2 won't work for you. I think 
Salvatore also alluded to that in his response. What you are suggesting 
with pinging the container from the appropriate namespace may be worth a 
try but then there may be containers that do not allow ingress traffic 
while they are up and happy. So short of what Salvatore suggested in his 
earlier email (and I am not sure if that can be done without additions to 
Neutron), we are left with option 1.

Keep in mind that users can choose not to enable the blocking option and 
things will be as they are right now. Would that be reasonable?

Best,

Mohammad

Neil Jerram ---06/10/2016 09:25:59 AM---Hi Mohammad, Why is the blocking 
needed? Is it to report some kind of status back to

From: Neil Jerram 
To: "OpenStack Development Mailing List (not for usage questions)" 

Date: 06/10/2016 09:25 AM
Subject: Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron Port 
is Active



Hi Mohammad,

Why is the blocking needed?  Is it to report some kind of status back to 
Docker/Kubernetes, or to allow some follow-on action to happen?

When using networking-calico as the driver, I think that only option (1) 
would work, out of the options you've suggested below.  (3) doesn't work, 
as you say, because Calico doesn't involve an L2 agent.  Also Calico 
doesn't use the RPC message queue for reporting port status, because we've 
found that that message queue is in itself a scalability bottleneck.

I guess another option would be for the using system to determine for 
itself when the port appears to be working, e.g. by the host pinging the 
container/pod's IP address.

Regards,
Neil


On Wed, Jun 8, 2016 at 4:23 PM Mohammad Banikazemi  wrote:
For the Kuryr project, in order to support blocking until vifs are plugged 
in (that is adding config options similar to the following options define 
in Nova: vif_plugging_is_fatal and vif_plugging_timeout), we need to 
detect that the Neutron plugin being used is done with plugging a given 
vif.

Here are a few options:

1- The simplest approach seems to be polling for the status of the Neutron 
port to become Active. (This may lead to scalability issues but short of 
having a specific goal for scalability, it is not clear that will be the 
case.)
2- Alternatively, We could subscribe to the message queue and wait for 
such a port update event. 
3- It was also suggested that we could use l2 agent extension to detect 
such an event but that seems to limit us to certain Neutron plugins and 
therefore not acceptable.

I was wondering if there are other and better options. 

Best,

Mohammad
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [neutron] [designate] multi-tenancy in Neutron's DNS integration

2016-05-09 Thread Mike Spreitzer
"Hayes, Graham" <graham.ha...@hpe.com> wrote on 05/09/2016 04:08:07 PM:

> ...
> On 09/05/2016 20:55, Mike Spreitzer wrote:
> ...
> > Oh, right, the network gets to specify the rest of the FQDN.  In my 
case
> > I am interested in Neutron Ports on tenant networks.  So with a 
per-port
> > "hostname" (first label) and per-network "domain" (rest of the 
labels),
> > I would get separation between tenants --- at least in the sense that
> > there is no overlap in FQDNs.  Will this work for private tenant 
networks?
> 
> Yes, you could publish the records to Designate for this, or using the
> internal dns resolution side of the integration.
> 
> Pushing the records to designate would make them viewable globally
> (anywhere the DNS servers are accessible)
> 
> 
> > The other part of separation is that I do not want one tenant to even 
be
> > able to look up FQDNs that belong to another tenant.  Is this
> > prohibition possible today?  If not, is anyone else interested in it?
> 
> Do you want to limit this to inside the tenant private network? if so, 
> just allowing users to set the dns_domain on a network, and not enabling 

> the external DNS plugin will work fine.

Ah, that may be what I want.  BTW, I am not planning to use Nova.  I am 
planning to use Swarm and Kubernetes to create containers attached to 
Neutron private tenant networks.  What DNS server would I configure those 
containers to use?

Thanks,
Mike




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] [designate] multi-tenancy in Neutron's DNS integration

2016-05-09 Thread Mike Spreitzer
"Hayes, Graham" <graham.ha...@hpe.com> wrote on 05/09/2016 03:00:34 PM:

> From: "Hayes, Graham" <graham.ha...@hpe.com>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Date: 05/09/2016 03:05 PM
> Subject: Re: [openstack-dev] [neutron] [designate] multi-tenancy in 
> Neutron's DNS integration
> 
> On 09/05/2016 19:21, Mike Spreitzer wrote:
> > I just read
> > 
http://docs.openstack.org/mitaka/networking-guide/adv-config-dns.htmland
> , unless
> > I missed something, it seems to be describing something that is not
> > multi-tenant.  I am focused on FQDNs for Neutron Ports.  For those, 
only
> > the "hostname" part (the first label, in official DNS jargon) is
> > controllable by the Neutron user, the rest of the FQDN is fixed in
> > Neutron configuration.  Have I got that right?  If so then I am
> > surprised.  I would have expected something that isolates tenants
> > (projects) from one another.  Is there any interest in such a thing?
> >
> > Thanks,
> > Mike
> 
> ...
> 
> If you have per-project networks the integration can be done on a
> project by project basis, with floating IPs assigned the name from
> the port and the zone from the private network.

Oh, right, the network gets to specify the rest of the FQDN.  In my case I 
am interested in Neutron Ports on tenant networks.  So with a per-port 
"hostname" (first label) and per-network "domain" (rest of the labels), I 
would get separation between tenants --- at least in the sense that there 
is no overlap in FQDNs.  Will this work for private tenant networks?

The other part of separation is that I do not want one tenant to even be 
able to look up FQDNs that belong to another tenant.  Is this prohibition 
possible today?  If not, is anyone else interested in it?

Thanks,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [neutron] [designate] multi-tenancy in Neutron's DNS integration

2016-05-09 Thread Mike Spreitzer
I just read 
http://docs.openstack.org/mitaka/networking-guide/adv-config-dns.html and, 
unless I missed something, it seems to be describing something that is not 
multi-tenant.  I am focused on FQDNs for Neutron Ports.  For those, only 
the "hostname" part (the first label, in official DNS jargon) is 
controllable by the Neutron user, the rest of the FQDN is fixed in Neutron 
configuration.  Have I got that right?  If so then I am surprised.  I 
would have expected something that isolates tenants (projects) from one 
another.  Is there any interest in such a thing?

Thanks,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [kuryr] Question for Antoni Puimedon about load balancers and overlay networks

2016-05-02 Thread Mike Spreitzer
I am looking at 
https://www.openstack.org/videos/video/project-kuryr-docker-delivered-kubernetes-next
 
, around 28:00.  You have said that overlay networks are involved, and are 
now talking about load balancers.  Is this Neutron LBaaS?  As far as I 
know, a Neutron LBaaS instance is "one-armed" --- both the VIP and the 
back endpoints have to be on the same Neutron network.  But you seem to 
have all the k8s services on a single subnet.  So I am having some trouble 
following exactly what is going on.  Can you please elaborate?

BTW, there is also some discussion of k8s multi-tenancy in the Kubernetes 
Networking SIG and the Kubernetes OpenStack SIG.

Thanks,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack][neutron] Eliminating the DevStack layer

2016-04-08 Thread Mike Spreitzer
I like this, for a reason not mentioned.  I am not a Neutron dev or 
operator and have never learned how to deploy Neutron; I have always 
driven it through DevStack.  The documentation for that has never been 
adequate, and I have concluded it will never be adequate.  With inadequate 
documentation, that layer isn't really doing its job anyway.  If there is 
no devstack layer getting in my way, I am willing to learn how to deploy 
Neutron from what I suppose is better documentation than I have been 
reading.  I understand that direct Neutron configuration is more 
troublesome than it should be.  Eliminating the devstack Neutron layer 
will only increase the pressure to both improve the documentation of 
Neutron configuration and simplify the subject, both of which are Good 
Things (TM).

Regards,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [kuryr] Did kuryr need to know about Docker's cluster store?

2016-03-03 Thread Mike Spreitzer
On Feb 5 I was given a tarchive of kuryr with an install script that 
configures the docker daemon to use consul as its cluster store.  If I 
modify the config of docker to use etcd instead then do I need to change 
anything in Kuryr?

Thanks,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] - Changing the Neutron default security group rules

2016-03-02 Thread Mike Spreitzer
Kevin Benton  wrote on 03/02/2016 01:27:27 PM:

> Does it at least also include the UUID, or is there no way to tell 
> from 'nova show'?

No direct way to tell, as far as I can see.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] - Changing the Neutron default security group rules

2016-03-02 Thread Mike Spreitzer
"Sean M. Collins"  wrote on 03/02/2016 01:16:52 PM:

> Meaning your users are creating new security groups and naming them
> "default" - so you have the "default" default (heh) and then the one
> that they created named default?
> 
> Are security group names in Nova-Net unqiue? I seem to recall that being
> a difference between Nova-Net and Neutron, where security group names
> are not unique in Neutron - hence the problem above.

I have seen this happen in a variety of use cases.  It does not bother me 
that security group names are scoped to tenants; other kinds of names also 
lack various kinds of uniqueness.  I am really just raising a tiny, 
peripheral rant here.  When I go in as admin to look at a problem, `nova 
show` identifies a Compute Instance's security group in a useless way.  If 
only I could make `nova show` tell me the UUID instead of, or in addition 
to, the security group's name then I would be happy.

Thanks,
Mike


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] - Changing the Neutron default security group rules

2016-03-02 Thread Mike Spreitzer
"Sean M. Collins"  wrote on 03/02/2016 12:38:29 PM:

> I think that the default security group should be left as is - and users
> should be trained that they should bring/create security groups with the
> appropriate rules for their need.

Could we at least make it less difficult to figure out which security 
group is attached to a Nova instance?  Right now `nova show` says only 
that the security group is named "default" and guess what --- they are 
*all* named default!  An admin looking at this is lost.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [devstack] [neutron] How to ask for linuxbridge instead of openvswitch

2016-02-29 Thread Mike Spreitzer
How would I modify the recipe at 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
 
to get linuxbridge instead of openvswitch?

Thanks,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] MTU configuration pain

2016-01-25 Thread Mike Spreitzer
BTW, regarding devstack: See 
https://bugs.launchpad.net/devstack/+bug/1532924.  I have been trying to 
get the current code to work, following the ideas in 
https://specs.openstack.org/openstack/fuel-specs/specs/7.0/jumbo-frames-between-instances.html#proposed-change
.  It fails only at the last step: the MTU on the network interface inside 
the VM is still 1500.

Regards,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] MTU configuration pain

2016-01-23 Thread Mike Spreitzer
Adam Lawson  wrote on 01/23/2016 02:27:46 PM:

> For the sake of over-simplification, is there ever a reason to NOT 
> enable jumbo frames in a cloud/SDN context where most of the traffic
> is between virtual elements that all support it? I understand that 
> some switches do not support it and traffic form the web doesn't 
> support it either but besides that, seems like a default 
> "jumboframes = 1" concept would work just fine to me.
> 
> Then again I'm all about making OpenStack easier to consume so my 
> ideas tend to gloss over special use cases with special requirements.

Regardless of the default, there needs to be clear documentation on what 
to do for those of us who can not use jumbo frames, and it needs to work. 
That goes for production deployers and also for developers using DevStack.

Thanks,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [devstack] [neutron] Progress, and current problems

2016-01-05 Thread Mike Spreitzer
I have been trying to get 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
 
working on Ubuntu 14.04 (updated to the latest, which includes kernel 
3.19).  With the latest sources and a sufficiently recent OVS, the 
one-node install pretty much works!  I added to local.conf only: (a) 
IP_VERSION=4, to avoid issues with v6, and (b) some settings to get logs 
saved to files without coloring.

In particular, what I have done regarding OVS is to download 2.4 and build 
the .deb packages according to the instructions.  I used the kernel 
datapath from the linux kernel 3.19.  I installed the openvswitch-common 
and openvswitch-switch .debs that I built.

I think the doc should be enhanced to discuss IPv6.  I tried once without 
the `IP_VERSION=4`, and the resulting IPv6 config was not good.

IMHO the default should be to keeps logs in files without coloring, 
because that is the most user-friendly.  I think that for DevStack the 
defaults should be friendly to newbies, since this is where they start.

I then went on to 
http://docs.openstack.org/developer/devstack/guides/neutron.html#adding-additional-compute-nodes
 
, and ran into MTU problems.  All MTUs come out to be 1500, which just 
does not work (due to the tunneling overhead).

I then tried a fresh DevStack install with a revised local.conf derived 
from 
https://specs.openstack.org/openstack/fuel-specs/specs/7.0/jumbo-frames-between-instances.html#proposed-change
 
.  Specifically, this is what I added to my local.conf for MTU issues:

[[post-config|$NEUTRON_CONF]]
[DEFAULT]
advertise_mtu=True
network_device_mtu=1450

[[post-config|/$Q_PLUGIN_CONF_FILE]]
[ml2]
physical_network_mtus = public:1450

That got me MTUs of 1450 on the router and DHCP ports but not the VM 
plumbing.  Perhaps I read too much into the word "keep".

Then I added

network_device_mtu = 65000

to the DEFAULT section of nova.conf, then restarted all the nova 
processes.  I created a new VM after that.  The result was that the 
qbr..., qvo..., qvb..., and tap... network interfaces in the main netns 
have MTU of 65000 --- and eth0 inside the VM has an MTU of 1500.

So then I changed the network_device_mtu setting in nova.conf to 1450, 
restarted all the nova processes again, and made another new VM.  Now the 
qbr..., qvo..., qvb..., and tap... network interfaces in the main netns 
have MTU of 1450 but the VM still got an MTU of 1500 on its eth0.

In my logs directory, `grep 'Running command' *.log | grep set | grep -i 
mtu` finds only the setting of the MTUs of the router and DHCP stuff --- 
nothing for any of the VMs.

Regards,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] [neutron] Procedure for back-porting a bug fix to stable/liberty branch of a Neutron script in DevStack?

2016-01-04 Thread Mike Spreitzer
> From: Kyle Mestery <mest...@mestery.com>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Date: 01/03/2016 02:15 PM
> Subject: Re: [openstack-dev] [devstack] [neutron] Procedure for 
> back-porting a bug fix to stable/liberty branch of a Neutron script 
> in DevStack?
> 
> On Sun, Jan 3, 2016 at 4:01 PM, Mike Spreitzer <mspre...@us.ibm.com> 
wrote:
> I would like to back-port https://review.openstack.org/#/c/242721/
> --- which fixed https://bugs.launchpad.net/devstack/+bug/1469596--- 
> to stable/liberty.  I have contributed to main line development 
> before, but not stable branches.  I see that http://
> docs.openstack.org/infra/manual/developers.htmldoes not specifically
> address this case.  What is the procedure to follow?

> The best place to look is in the project team guide [1], 
> specifically the section on proposing fixes.
> 
> [1] http://docs.openstack.org/project-team-guide/stable-branches.html
> [2] 
http://docs.openstack.org/project-team-guide/stable-branches.html#proposing-fixes


Reference [2] says I should

$ git checkout stable/tango

but that doesn't work.  Here is a typescript of how it goes wrong:

mjs10:devstack mspreitz$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

mjs10:devstack mspreitz$ git pull
Already up-to-date.

mjs10:devstack mspreitz$ git remote -v
gerrit 
ssh://mspre...@review.openstack.org:29418/openstack-dev/devstack.git 
(fetch)
gerrit 
ssh://mspre...@review.openstack.org:29418/openstack-dev/devstack.git 
(push)
origin  https://github.com/openstack-dev/devstack.git (fetch)
origin  https://github.com/openstack-dev/devstack.git (push)

mjs10:devstack mspreitz$ git checkout stable/liberty
error: pathspec 'stable/liberty' did not match any file(s) known to git.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] [neutron] Procedure for back-porting a bug fix to stable/liberty branch of a Neutron script in DevStack?

2016-01-03 Thread Mike Spreitzer
> From: Kyle Mestery <mest...@mestery.com>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Date: 01/03/2016 02:15 PM
> Subject: Re: [openstack-dev] [devstack] [neutron] Procedure for 
> back-porting a bug fix to stable/liberty branch of a Neutron script 
> in DevStack?
> 
> On Sun, Jan 3, 2016 at 4:01 PM, Mike Spreitzer <mspre...@us.ibm.com> 
wrote:
> I would like to back-port https://review.openstack.org/#/c/242721/
> --- which fixed https://bugs.launchpad.net/devstack/+bug/1469596--- 
> to stable/liberty.  I have contributed to main line development 
> before, but not stable branches.  I see that http://
> docs.openstack.org/infra/manual/developers.htmldoes not specifically
> address this case.  What is the procedure to follow?

> The best place to look is in the project team guide [1], 
> specifically the section on proposing fixes.
> 
> [1] http://docs.openstack.org/project-team-guide/stable-branches.html
> [2] http://docs.openstack.org/project-team-guide/stable-
> branches.html#proposing-fixes

Regarding the mechanics, I was given the following alternate recipe in a 
discussion on #openstack-neutron; I assume it is pretty much equivalent.

"do something like 'git checkout -b libery/backport/### 
remotes/origin/stable/liberty' then 'git pull' then 'git review -X ###' 
then push it for review"

Thanks,
Mike


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [devstack] [neutron] Procedure for back-porting a bug fix to stable/liberty branch of a Neutron script in DevStack?

2016-01-03 Thread Mike Spreitzer
I would like to back-port https://review.openstack.org/#/c/242721/ --- 
which fixed https://bugs.launchpad.net/devstack/+bug/1469596 --- to 
stable/liberty.  I have contributed to main line development before, but 
not stable branches.  I see that 
http://docs.openstack.org/infra/manual/developers.html does not 
specifically address this case.  What is the procedure to follow?

Thanks,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] Multiple locations for documentation of features

2015-12-08 Thread Mike Spreitzer
> From: Henry Gessau 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: 12/04/2015 02:23 PM
> Subject: Re: [openstack-dev] [neutron] Multiple locations for 
> documentation of features
> 
> Sean M. Collins  wrote:
> > I've noticed that a lot of features are now being documented as RSTs
> > inside of devref. Like the following:
> > 
> > https://review.openstack.org/#/c/251859/
> > 
> > But there are lots already present. Can someone point out to me what 
the
> > criteria is for these documents? I am a little confused about the
> > relationship between neutron-specs, RFE bugs, and some features being
> > documented in devref. Especially when a review includes the actual 
code,
> > then a new RST file in devref - wasn't that what specs were for?
> 
> Here is how I would like to see things ending up:
> 
> 1. RFE: "I want X"
> 2. Spec: "I plan to implement X like this"
> 3. devref: "How X is implemented and how to extend it"
> 4. OS docs: "API and guide for using X"
> 
> Once X is implemented I don't want to have to go to 1 or 2 to find 
information
> on it. The devref may have a lot of content from the spec, but the spec 
is not
> maintained and the implementation may differ in some ways. The devref 
should
> be kept current with refactorings, etc. of the implementation.

Lots cheers from me too.  Let me add one thing: "the spec is not 
maintained" is a remaining process bug.  A spec by itself is a very useful 
thing.  It is the first thing to read when trying to understand the 
implementation.  How about we resolve that devref includes a maintained 
version of the spec?

Regards,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] The OVS and Provider Networks section of the guide to Neutron in DevStack

2015-10-20 Thread Mike Spreitzer
First let me make sure I understand what this section (
http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks
) is trying to say.  The second paragraph is saying that some 
infrastructure provider has allocated a VLAN tag (and the later display of 
localrc contents uses 2010 as that tag) and an address range (later seen 
to be 203.0.113.0/24) to use on that VLAN on provider_net.  The exhibited 
localrc contents also say that tenant networks can be created, using VLAN 
tags drawn from the range 3001:4000.  Those localrc contents do not 
explicitly say which ethernet --- provider_net or control_plane --- will 
carry those tenant VLANs.  Which ethernet do you think those localrc 
contents imply will carry the tenant VLANs?

Second, I tested this using stable/liberty on Ubuntu 14.04.03, and got a 
few surprises.  I did exactly as described by that section, starting from 
a "physical network setup" as described there (but it was actually 
virtual, using VirtualBox), and using the localrc contents exhibited there 
except patched to fix bugs 
https://bugs.launchpad.net/devstack/+bug/1508195 and 
https://bugs.launchpad.net/devstack/+bug/1508202 (the latter by sadly 
stipulating IP_VERSION=4).  After creating the controller, compute1, and 
compute2 I looked at `ovs-vsctl show` on the controller, and found that 
br-tun had two VxLan ports (see display below).  This surprised me because 
I thought that VxLan can itself handle multiple hosts, it is not just 
point-to-point.

Bridge br-tun
fail_mode: secure
Port "vxlan-0a03"
Interface "vxlan-0a03"
type: vxlan
options: {df_default="true", in_key=flow, 
local_ip="10.0.0.2", out_key=flow, remote_ip="10.0.0.3"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "vxlan-0a04"
Interface "vxlan-0a04"
type: vxlan
options: {df_default="true", in_key=flow, 
local_ip="10.0.0.2", out_key=flow, remote_ip="10.0.0.4"}
Port br-tun
Interface br-tun
type: internal

My second surprise is that the path from controller to the VM's host goes 
over the control_plane network --- somewhat contrary to its name!

My third surprise came after I made a VM on the provider network and tried 
to ping it from the router.  I got no response.  A little investigation 
shows that the router's ARP request for the VM gets lost somewhere between 
br-int and br-tun.  `tcpdump -nne -i br-int` on the controller shows the 
ARP requests appearing with VLAN tag 1 (which is right, the external VLAN 
tags are translated to internal host-local tags on br-int), and no ARP 
replies.  OTOH, `tcpdump -nne -i br-tun` does not even find the ARP 
requests.

Digging a little deeper, I looked at the OpenFlow tables on the 
controller.  It looks like the flow table for br-tun does indeed prescribe 
that all traffic coming from outside be dropped!  Here is the output from 
`ovs-appctl bridge/dump-flows br-tun`:

duration=23916s, priority=1, n_packets=0, n_bytes=0, 
priority=1,in_port=3,actions=resubmit(,4)
duration=39904s, priority=1, n_packets=82, n_bytes=7025, 
priority=1,in_port=1,actions=resubmit(,2)
duration=36711s, priority=1, n_packets=0, n_bytes=0, 
priority=1,in_port=2,actions=resubmit(,4)
duration=39904s, priority=0, n_packets=7, n_bytes=558, 
priority=0,actions=drop
table_id=2, duration=39904s, priority=0, n_packets=0, n_bytes=0, 
priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00,actions=resubmit(,20)
table_id=2, duration=39904s, priority=0, n_packets=82, n_bytes=7025, 
priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00,actions=resubmit(,22)
table_id=3, duration=39904s, priority=0, n_packets=0, n_bytes=0, 
priority=0,actions=drop
table_id=4, duration=39904s, priority=0, n_packets=0, n_bytes=0, 
priority=0,actions=drop
table_id=6, duration=39904s, priority=0, n_packets=0, n_bytes=0, 
priority=0,actions=drop
table_id=10, duration=39904s, priority=1, n_packets=0, n_bytes=0, 
priority=1,actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xb64bc7164f7e1137,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
table_id=20, duration=39904s, priority=0, n_packets=0, n_bytes=0, 
priority=0,actions=resubmit(,22)
table_id=22, duration=39904s, priority=0, n_packets=82, n_bytes=7025, 
priority=0,actions=drop
table_id=254, duration=39904s, priority=0, n_packets=0, n_bytes=0, 
priority=0,reg0=0x3,actions=drop
table_id=254, duration=39904s, priority=0, n_packets=1, n_bytes=90, 
priority=0,reg0=0x1,actions=controller(reason=no_match)
table_id=254, duration=39904s, priority=0, n_packets=0, n_bytes=0, 
priority=0,reg0=0x2,actions=drop

By examining `ovs-vsctl list Interface` I can see that patch-int is 
OpenFlow 

Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron

2015-10-17 Thread Mike Spreitzer
"Sean M. Collins" <s...@coreitpro.com> wrote on 10/16/2015 01:03:53 PM:

> From: "Sean M. Collins" <s...@coreitpro.com>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Date: 10/16/2015 01:12 PM
> Subject: Re: [openstack-dev] [devstack] [neutron] A larger batch of 
> questions about configuring DevStack to use Neutron
> 
> On Fri, Oct 16, 2015 at 02:02:57AM EDT, Mike Spreitzer wrote:
> > Now I have tested the first section (taking the latest of both 
changes) 
> > using stable/liberty, and it failed because br-ex was not created 
> > automatically.  I opened 
https://bugs.launchpad.net/devstack/+bug/1506733
> > 
> > I think Kilo did not have this problem.
> > 
> 
> Ok. I have assigned myself the bug. I will be sightseeing with my 
partner
> Caroline in Tokyo next week before the summit, so it may take me a
> little bit of time to get around to it, but I will try and follow up on
> the bug after the summit.

Thanks.  I am going to do some more tests in the interim, to see what I 
can add.

Thanks,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron

2015-10-16 Thread Mike Spreitzer
> From: Mike Spreitzer/Watson/IBM@IBMUS
> To: "OpenStack Development Mailing List \(not for usage questions\)"
> <openstack-dev@lists.openstack.org>
> Date: 10/13/2015 03:39 AM
> Subject: Re: [openstack-dev] [devstack] [neutron] A larger batch of 
> questions about configuring DevStack to use Neutron
> 
> > From: "Sean M. Collins" <s...@coreitpro.com>
> > To: "OpenStack Development Mailing List (not for usage questions)" 
> > <openstack-dev@lists.openstack.org>
> > Date: 10/12/2015 11:34 AM
> > Subject: Re: [openstack-dev] [devstack] [neutron] A larger batch of 
> > questions about configuring DevStack to use Neutron
> > 
> > On Thu, Oct 08, 2015 at 01:47:31PM EDT, Mike Spreitzer wrote:
> > > ..
> > > > > In the section 
> > > > > http://docs.openstack.org/developer/devstack/guides/
> > > > neutron.html#using-neutron-with-a-single-interface
> > > > > there is a helpful display of localrc contents.  It says, among 
other 
> > > > > things,
> > > > > 
> > > > >OVS_PHYSICAL_BRIDGE=br-ex
> > > > >PUBLIC_BRIDGE=br-ex
> > > > > 
> > > > > In the next top-level section, 
> > > > > http://docs.openstack.org/developer/devstack/guides/
> > > > neutron.html#using-neutron-with-multiple-interfaces
> > > > > , there is no display of revised localrc contents and no mention 
of 
> > > > > changing either bridge setting.  That is an oversight, right?
> > > > 
> > > > No, this is deliberate. Each section is meant to be independent, 
since
> > > > each networking configuration and corresponding DevStack 
configuration
> > > > is different. Of course, this may need to be explicitly stated in 
the
> > > > guide, so there is always room for improvement.
> > > 
> > > I am not quite sure I understand your answer.  Is the intent that I 
can 
> > > read only one section, ignore all the others, and that will tellme 
how to 
> > > use DevStack to produce that section's configuration?  If so 
> then it would 
> > > be good if each section had a display of all the necessary localrc 
> > > contents.
> > 
> > Agreed. This is a failure on my part, because I was pasting in only
> > parts of my localrc (since it came out of a live lab environment). 
I've
> > started pushing patches to correct this.
> > 
> > 
> > > I have started over, from exactly the picture drawn at the start of 
the 
> > > doc.  That has produced a configuration that mostly works.  However, 
I 
> > > tried creating a VM connected to the public network, and that failed 
for 
> > > lack of a Neutron DHCP server there.
> > 
> > The public network is used for floating IPs. The L3 agent creates NAT
> > rules to intercept traffic on the public network and NAT it back to a
> > guest instance that has the floating IP allocated to it.
> > 
> > The behavior for when a guest is directly attached to the public
> > network, I sort of forget what happens exactly but I do know that it
> > doesn't work from experience - most likely because the network is not
> > configured as a flat network. It will not receive a DHCP lease from 
the
> > external router.
> > 
> > > I am going to work out how to change 
> > > that, and am willing to contribute an update to this doc.  Wouldyou 
want 
> > > that in this section --- in which case this section needs to specify 
that 
> > > the provider DOES NOT already have DHCP service on the hardware 
network 
> > > --- or as a new section?
> > 
> > No, I think we should maybe have a warning or something that the
> > external network will be used for Floating IPs, and that guest 
instances
> > should not be directly attached to that network.
> > 
> > > > > 
> > > > > Looking at 
> > > > > 
http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html(or
> > > , in 
> > > > > former days, the doc now preserved at 
> > > > > http://docs.ocselected.org/openstack-manuals/kilo/networking-
> > > > guide/content/under_the_hood_openvswitch.html
> > > > > ) I see the name br-ex used for $PUBLIC_BRIDGE--- not 
> > > $OVS_PHYSICAL_BRIDGE
> > > > > , right?  Wouldn't it be less confusing if 
> > > > > 
http://docs.openstack.org/developer/devstack/guides/neutron.htmluseda 
> > > 
> > > > > name other than "br-ex" 

Re: [openstack-dev] [devstack] A few questions on configuring DevStack for Neutron

2015-10-14 Thread Mike Spreitzer
Matt Riedemann  wrote on 10/08/2015 09:48:33 
PM:

> On 10/8/2015 11:38 AM, Sean M. Collins wrote:
> > Please see my response here:
> >
> > 
http://lists.openstack.org/pipermail/openstack-dev/2015-October/076251.html

> >
> > In the future, do not create multiple threads since responses will get
> > lost
> >
> 
> Maybe a dumb question, but couldn't people copy the localrc from the 
> gate-tempest-dsvm-neutron-full job, i.e.:
> 
> http://logs.openstack.org/20/231720/1/check/gate-tempest-dsvm-
> neutron-full/f418dc8/logs/localrc.txt.gz

Matt, thanks for the reminder.  I was pointed at one of those once before, 
but do not remember how to find them in general.  To be useful in 
http://docs.openstack.org/developer/devstack/guides/neutron.html we need 
to identify which section they illuminate, or add another section with 
appropriate explanation.  Which would it be?

Sean, you said that those URLs are tribal knowledge.  Would you recommend 
documenting them and, if so, where?  I see that the one Matt cited is part 
of a comprehensive archive from a tempest run, and there is even some 
explanatory material included within that run's archive.  Would it be 
possible and appropriate for the DevStack Neutron guide to point to some 
documentation that describes these archives?

Thanks,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron

2015-10-13 Thread Mike Spreitzer
> From: "Sean M. Collins" <s...@coreitpro.com>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Date: 10/12/2015 11:34 AM
> Subject: Re: [openstack-dev] [devstack] [neutron] A larger batch of 
> questions about configuring DevStack to use Neutron
> 
> On Thu, Oct 08, 2015 at 01:47:31PM EDT, Mike Spreitzer wrote:
> > ..
> > > > In the section 
> > > > http://docs.openstack.org/developer/devstack/guides/
> > > neutron.html#using-neutron-with-a-single-interface
> > > > there is a helpful display of localrc contents.  It says, among 
other 
> > > > things,
> > > > 
> > > >OVS_PHYSICAL_BRIDGE=br-ex
> > > >PUBLIC_BRIDGE=br-ex
> > > > 
> > > > In the next top-level section, 
> > > > http://docs.openstack.org/developer/devstack/guides/
> > > neutron.html#using-neutron-with-multiple-interfaces
> > > > , there is no display of revised localrc contents and no mention 
of 
> > > > changing either bridge setting.  That is an oversight, right?
> > > 
> > > No, this is deliberate. Each section is meant to be independent, 
since
> > > each networking configuration and corresponding DevStack 
configuration
> > > is different. Of course, this may need to be explicitly stated in 
the
> > > guide, so there is always room for improvement.
> > 
> > I am not quite sure I understand your answer.  Is the intent that I 
can 
> > read only one section, ignore all the others, and that will tell me 
how to 
> > use DevStack to produce that section's configuration?  If so then it 
would 
> > be good if each section had a display of all the necessary localrc 
> > contents.
> 
> Agreed. This is a failure on my part, because I was pasting in only
> parts of my localrc (since it came out of a live lab environment). I've
> started pushing patches to correct this.
> 
> 
> > I have started over, from exactly the picture drawn at the start of 
the 
> > doc.  That has produced a configuration that mostly works.  However, I 

> > tried creating a VM connected to the public network, and that failed 
for 
> > lack of a Neutron DHCP server there.
> 
> The public network is used for floating IPs. The L3 agent creates NAT
> rules to intercept traffic on the public network and NAT it back to a
> guest instance that has the floating IP allocated to it.
> 
> The behavior for when a guest is directly attached to the public
> network, I sort of forget what happens exactly but I do know that it
> doesn't work from experience - most likely because the network is not
> configured as a flat network. It will not receive a DHCP lease from the
> external router.
> 
> > I am going to work out how to change 
> > that, and am willing to contribute an update to this doc.  Would you 
want 
> > that in this section --- in which case this section needs to specify 
that 
> > the provider DOES NOT already have DHCP service on the hardware 
network 
> > --- or as a new section?
> 
> No, I think we should maybe have a warning or something that the
> external network will be used for Floating IPs, and that guest instances
> should not be directly attached to that network.
> 
> > > > 
> > > > Looking at 
> > > > 
http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html(or
> > , in 
> > > > former days, the doc now preserved at 
> > > > http://docs.ocselected.org/openstack-manuals/kilo/networking-
> > > guide/content/under_the_hood_openvswitch.html
> > > > ) I see the name br-ex used for $PUBLIC_BRIDGE--- not 
> > $OVS_PHYSICAL_BRIDGE
> > > > , right?  Wouldn't it be less confusing if 
> > > > 
http://docs.openstack.org/developer/devstack/guides/neutron.htmlused a 
> > 
> > > > name other than "br-ex" for the exhibited commands that apply to 
> > > > $OVS_PHYSICAL_BRIDGE?
> > > 
> > > No, this is deliberate - br-ex is the bridge that is used for 
external
> > > network traffic - such as floating IPs and public IP address ranges. 
On
> > > the network node, a physical interface is attached to br-ex so that
> > > traffic will flow.
> > > 
> > > PUBLIC_BRIDGE is a carryover from DevStack's Nova-Network support 
and is
> > > used in some places, with OVS_PHYSICAL_BRIDGE being used by 
DevStack's
> > > Neutron support, for the Open vSwitch driver specifically. They are 
two
> > > variables that for the most part serve the same pu

Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron

2015-10-12 Thread Mike Spreitzer
Thanks, i will review soon.  BTW, i am interested in creating a config in which 
a Compute Instance can be created on an external network.

Thanks,
Mike




   Sean M. Collins --- Re: [openstack-dev] [devstack] [neutron] A larger batch 
of questions about configuring DevStack to use Neutron --- 
From:"Sean M. Collins" <s...@coreitpro.com>To:"OpenStack Development 
Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>Date:Mon, Oct 12, 2015 11:34Subject:Re: 
[openstack-dev] [devstack] [neutron] A larger batch of questions about 
configuring DevStack to use Neutron
  On Thu, Oct 08, 2015 at 01:47:31PM EDT, Mike Spreitzer wrote: > .. > > > In 
the section > > > http://docs.openstack.org/developer/devstack/guides/ > > 
neutron.html#using-neutron-with-a-single-interface > > > there is a helpful 
display of localrc contents. It says, among other > > > things, > > > > > > 
OVS_PHYSICAL_BRIDGE=br-ex > > > PUBLIC_BRIDGE=br-ex > > > > > > In the next 
top-level section, > > > http://docs.openstack.org/developer/devstack/guides/ > 
> neutron.html#using-neutron-with-multiple-interfaces > > > , there is no 
display of revised localrc contents and no mention of > > > changing either 
bridge setting. That is an oversight, right? > > > > No, this is deliberate. 
Each section is meant to be independent, since > > each networking 
configuration and corresponding DevStack configuration > > is different. Of 
course, this may need to be explicitly stated in the > > guide, so there is 
always room for improvement. > > I am not quite sure I understand your answer. 
Is the intent that I can > read only one section, ignore all the others, and 
that will tell me how to > use DevStack to produce that section's 
configuration? If so then it would > be good if each section had a display of 
all the necessary localrc > contents. Agreed. This is a failure on my part, 
because I was pasting in only parts of my localrc (since it came out of a live 
lab environment). I've started pushing patches to correct this. > I have 
started over, from exactly the picture drawn at the start of the > doc. That 
has produced a configuration that mostly works. However, I > tried creating a 
VM connected to the public network, and that failed for > lack of a Neutron 
DHCP server there. The public network is used for floating IPs. The L3 agent 
creates NAT rules to intercept traffic on the public network and NAT it back to 
a guest instance that has the floating IP allocated to it. The behavior for 
when a guest is directly attached to the public network, I sort of forget what 
happens exactly but I do know that it doesn't work from experience - most 
likely because the network is not configured as a flat network. It will not 
receive a DHCP lease from the external router. > I am going to work out how to 
change > that, and am willing to contribute an update to this doc. Would you 
want > that in this section --- in which case this section needs to specify 
that > the provider DOES NOT already have DHCP service on the hardware network 
> --- or as a new section? No, I think we should maybe have a warning or 
something that the external network will be used for Floating IPs, and that 
guest instances should not be directly attached to that network. > > > > > > 
Looking at > > > 
http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html(or > , in > 
> > former days, the doc now preserved at > > > 
http://docs.ocselected.org/openstack-manuals/kilo/networking- > > 
guide/content/under_the_hood_openvswitch.html > > > ) I see the name br-ex used 
for $PUBLIC_BRIDGE--- not > $OVS_PHYSICAL_BRIDGE > > > , right? Wouldn't it be 
less confusing if > > > 
http://docs.openstack.org/developer/devstack/guides/neutron.htmlused a > > > > 
name other than "br-ex" for the exhibited commands that apply to > > > 
$OVS_PHYSICAL_BRIDGE? > > > > No, this is deliberate - br-ex is the bridge that 
is used for external > > network traffic - such as floating IPs and public IP 
address ranges. On > > the network node, a physical interface is attached to 
br-ex so that > > traffic will flow. > > > > PUBLIC_BRIDGE is a carryover from 
DevStack's Nova-Network support and is > > used in some places, with 
OVS_PHYSICAL_BRIDGE being used by DevStack's > > Neutron support, for the Open 
vSwitch driver specifically. They are two > > variables that for the most part 
serve the same purpose. Frankly, > > DevStack has a lot of problems with 
configuration knobs, and > > PUBLIC_BRIDGE and OVS_PHYSICAL_BRIDGE is just a 
symptom. > > 

Re: [openstack-dev] [all] Recording little everyday OpenStack successes

2015-10-09 Thread Mike Spreitzer
Thierry Carrez  wrote on 10/09/2015 05:42:49 AM:

...
> So whenever you feel like you made progress, or had a little success in
> your OpenStack adventures, or have some joyful moment to share, just
> throw the following message on your local IRC channel:
> 
> #success [Your message here]
> 
> The openstackstatus bot will take that and record it on this wiki page:
> 
> https://wiki.openstack.org/wiki/Successes
> 
> We'll add a few of those every week to the weekly newsletter (as part of
> the developer digest that we reecently added there).
> 
> Caveats: Obviously that only works on channels where openstackstatus is
> present (the official OpenStack IRC channels), and we may remove entries
> that are off-topic or spam.
> 
> So... please use #success liberally and record lttle everyday OpenStack
> successes. Share the joy and make the OpenStack community a happy place.

Great.  I am about to contribute one myself.  Lucky I noticed this email. 
How will the word get out to those who did not?  How about a pointer to 
instructions on the Successes page?

Thanks,
Mike


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron

2015-10-08 Thread Mike Spreitzer
> From: "Sean M. Collins" <s...@coreitpro.com>
...
> 
> On Tue, Oct 06, 2015 at 11:25:03AM EDT, Mike Spreitzer wrote:
> > [Sorry, but I do not know if the thundering silence is because these 
> > questions are too hard, too easy, grossly off-topic, or simply because 

> > nobody cares.]
> 
> You sent your first e-mail on a Saturday. I saw it and flagged it for
> reply, but have not had a chance yet. It's only Tuesday. I do care and
> your questions are important. I will say though that it's a little
> difficult to answer your e-mail because of formatting and your thoughts
> seem to jump around. This is not intended as a personal criticism, it's
> just a little difficult to follow your e-mail in order to reply.

Thanks, I am glad somebody cares.  I used different subject lines because 
I was suspecting that I did not flag them correctly.  I see now that I was 
just too impatient.

..
> > In the section 
> > http://docs.openstack.org/developer/devstack/guides/
> neutron.html#using-neutron-with-a-single-interface
> > there is a helpful display of localrc contents.  It says, among other 
> > things,
> > 
> >OVS_PHYSICAL_BRIDGE=br-ex
> >PUBLIC_BRIDGE=br-ex
> > 
> > In the next top-level section, 
> > http://docs.openstack.org/developer/devstack/guides/
> neutron.html#using-neutron-with-multiple-interfaces
> > , there is no display of revised localrc contents and no mention of 
> > changing either bridge setting.  That is an oversight, right?
> 
> No, this is deliberate. Each section is meant to be independent, since
> each networking configuration and corresponding DevStack configuration
> is different. Of course, this may need to be explicitly stated in the
> guide, so there is always room for improvement.

I am not quite sure I understand your answer.  Is the intent that I can 
read only one section, ignore all the others, and that will tell me how to 
use DevStack to produce that section's configuration?  If so then it would 
be good if each section had a display of all the necessary localrc 
contents.  OTOH, if some sections build on other sections, it would be 
good if the dependent sections display the localrc contents that differ. 
Right now, the section at 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces
 
does not display any localrc contents at all.

> For example, There needs
> to be some editing done for that doc - the part about disabling the
> firewall is just dropped in the middle of the doc and breaks the flow -
> among other things. This is obviously not helpful to a new reader and we
> need to fix.
> 
> 
> > I am 
> > guessing I need to set OVS_PHYSICAL_BRIDGEand PUBLIC_BRIDGEto 
different 
> > values, and the exhibited `ovs-vsctl` commands in this section apply 
to 
> > $OVS_PHYSICAL_BRIDGE.  Is that right?  Are there other revisions I 
need to 
> > make to localrc?
> 
> No, this is not correct.
> 
> What does your networking layout look like on the DevStack node that you
> are trying to configure?

I have started over, from exactly the picture drawn at the start of the 
doc.  That has produced a configuration that mostly works.  However, I 
tried creating a VM connected to the public network, and that failed for 
lack of a Neutron DHCP server there.  I am going to work out how to change 
that, and am willing to contribute an update to this doc.  Would you want 
that in this section --- in which case this section needs to specify that 
the provider DOES NOT already have DHCP service on the hardware network 
--- or as a new section?

> 
> 
> > 
> > Looking at 
> > http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html(or
, in 
> > former days, the doc now preserved at 
> > http://docs.ocselected.org/openstack-manuals/kilo/networking-
> guide/content/under_the_hood_openvswitch.html
> > ) I see the name br-ex used for $PUBLIC_BRIDGE--- not 
$OVS_PHYSICAL_BRIDGE
> > , right?  Wouldn't it be less confusing if 
> > http://docs.openstack.org/developer/devstack/guides/neutron.htmlused a 

> > name other than "br-ex" for the exhibited commands that apply to 
> > $OVS_PHYSICAL_BRIDGE?
> 
> No, this is deliberate - br-ex is the bridge that is used for external
> network traffic - such as floating IPs and public IP address ranges. On
> the network node, a physical interface is attached to br-ex so that
> traffic will flow.
> 
> PUBLIC_BRIDGE is a carryover from DevStack's Nova-Network support and is
> used in some places, with OVS_PHYSICAL_BRIDGE being used by DevStack's
> Neutron support, for the Open vSwitch driver specifically. They are two
> variables that for the most part serve the same p

[openstack-dev] [devstack] [neutron] A larger batch of questions about configuring DevStack to use Neutron

2015-10-06 Thread Mike Spreitzer
[Sorry, but I do not know if the thundering silence is because these 
questions are too hard, too easy, grossly off-topic, or simply because 
nobody cares.]

I have been looking at 
http://docs.openstack.org/developer/devstack/guides/neutron.htmland wonder 
about a few things.

In the section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
there is a helpful display of localrc contents.  It says, among other 
things,

   OVS_PHYSICAL_BRIDGE=br-ex
   PUBLIC_BRIDGE=br-ex

In the next top-level section, 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces
, there is no display of revised localrc contents and no mention of 
changing either bridge setting.  That is an oversight, right?  I am 
guessing I need to set OVS_PHYSICAL_BRIDGEand PUBLIC_BRIDGEto different 
values, and the exhibited `ovs-vsctl` commands in this section apply to 
$OVS_PHYSICAL_BRIDGE.  Is that right?  Are there other revisions I need to 
make to localrc?

Looking at 
http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html(or, in 
former days, the doc now preserved at 
http://docs.ocselected.org/openstack-manuals/kilo/networking-guide/content/under_the_hood_openvswitch.html
) I see the name br-ex used for $PUBLIC_BRIDGE--- not $OVS_PHYSICAL_BRIDGE
, right?  Wouldn't it be less confusing if 
http://docs.openstack.org/developer/devstack/guides/neutron.htmlused a 
name other than "br-ex" for the exhibited commands that apply to 
$OVS_PHYSICAL_BRIDGE?

The section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch
builds on 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces
NOT 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
--- right?  Could I stop after reading that section, or must I go on to 
http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks
?

The exhibited localrc contents in section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
include both of these:

   Q_L3_ENABLED=True
   Q_USE_PROVIDERNET_FOR_PUBLIC=True

and nothing gainsays either of them until section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks
--- where we first see

   Q_L3_ENABLED=False

Is it true that all the other sections want both Q_L3_ENABLEDand 
Q_USE_PROVIDERNET_FOR_PUBLICto be True?

I tried adding IPv6 support to the recipe of the first section (
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
).  I added this to my localrc:

IP_VERSION=4+6
IPV6_PUBLIC_RANGE=fddf:2::/64
IPV6_PUBLIC_NETWORK_GATEWAY=fddf:2::1
IPV6_ROUTER_GW_IP=fddf:2::231

At first I had tried setting a different set of IPv6 variables (having 
only IP_VERSION in common with what I exhibit here), but found those: (a) 
duplicated the defaults and (b) caused problems due to lack of the ones I 
mention here.  Even the ones mentioned here led to a problem.  There is a 
bit of scripging that replaces my setting for IPV6_ROUTER_GW_IP with 
something dug out of Neutron.  That went wrong.  It replaced my setting 
with fddf:2::2, but that address was already in use by something else.

Thanks,
Mike


Thanks,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [devstack] A few questions on configuring DevStack for Neutron

2015-10-04 Thread Mike Spreitzer
[Apologies for re-posting, but I botched the subject line the first time 
and know that people use filters.]

I have been looking at 
http://docs.openstack.org/developer/devstack/guides/neutron.html and 
wonder about a few things.

In the section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
 
there is a helpful display of localrc contents.  It says, among other 
things,

OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_BRIDGE=br-ex

In the next top-level section, 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces
, there is no display of revised localrc contents and no mention of 
changing either bridge setting.  That is an oversight, right?  I am 
guessing I need to set OVS_PHYSICAL_BRIDGEand PUBLIC_BRIDGEto different 
values, and the exhibited `ovs-vsctl` commands in this section apply to 
$OVS_PHYSICAL_BRIDGE.  Is that right?  Are there other revisions I need to 
make to localrc?

Looking at 
http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html (or, 
in former days, the doc now preserved at 
http://docs.ocselected.org/openstack-manuals/kilo/networking-guide/content/under_the_hood_openvswitch.html
) I see the name br-ex used for $PUBLIC_BRIDGE--- not $OVS_PHYSICAL_BRIDGE
, right?  Wouldn't it be less confusing if 
http://docs.openstack.org/developer/devstack/guides/neutron.html used a 
name other than "br-ex" for the exhibited commands that apply to 
$OVS_PHYSICAL_BRIDGE?

The section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch
 
builds on 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces
 
NOT 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
 
--- right?  Could I stop after reading that section, or must I go on to 
http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks
 
?

The exhibited localrc contents in section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
 
include both of these:

Q_L3_ENABLED=True
Q_USE_PROVIDERNET_FOR_PUBLIC=True

and nothing gainsays either of them until section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks
 
--- where we first see

Q_L3_ENABLED=False

Is it true that all the other sections want both Q_L3_ENABLEDand 
Q_USE_PROVIDERNET_FOR_PUBLICto be True?

Thanks,
Mike


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] A few questions on configuring DevStack for Neutron

2015-10-03 Thread Mike Spreitzer
I have been looking at 
http://docs.openstack.org/developer/devstack/guides/neutron.html and 
wonder about a few things.

In the section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
 
there is a helpful display of localrc contents.  It says, among other 
things,

OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_BRIDGE=br-ex

In the next top-level section, 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces
 
, there is no display of revised localrc contents and no mention of 
changing either bridge setting.  That is an oversight, right?  I am 
guessing I need to set OVS_PHYSICAL_BRIDGE and PUBLIC_BRIDGE to different 
values, and the exhibited `ovs-vsctl` commands in this section apply to 
$OVS_PHYSICAL_BRIDGE.  Is that right?  Are there other revisions I need to 
make to localrc?

Looking at 
http://docs.openstack.org/networking-guide/scenario_legacy_ovs.html (or, 
in former days, the doc now preserved at 
http://docs.ocselected.org/openstack-manuals/kilo/networking-guide/content/under_the_hood_openvswitch.html
) I see the name br-ex used for $PUBLIC_BRIDGE --- not $
OVS_PHYSICAL_BRIDGE, right?  Wouldn't it be less confusing if 
http://docs.openstack.org/developer/devstack/guides/neutron.html used a 
name other than "br-ex" for the exhibited commands that apply to 
$OVS_PHYSICAL_BRIDGE?

The section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch
 
builds on 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-multiple-interfaces
 
NOT 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
 
--- right?  Could I stop after reading that section, or must I go on to 
http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks
 
?

The exhibited localrc contents in section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface
 
include both of these:

Q_L3_ENABLED=True
Q_USE_PROVIDERNET_FOR_PUBLIC=True

and nothing gainsays either of them until section 
http://docs.openstack.org/developer/devstack/guides/neutron.html#neutron-networking-with-open-vswitch-and-provider-networks
 
--- where we first see

Q_L3_ENABLED=False

Is it true that all the other sections want both Q_L3_ENABLED and 
Q_USE_PROVIDERNET_FOR_PUBLIC to be True ?

Thanks,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] -1 due to line length violation in commit messages

2015-09-30 Thread Mike Spreitzer
> From: Gorka Eguileor 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: 09/29/2015 07:34 AM
> Subject: Re: [openstack-dev] [all] -1 due to line length violation 
> in commit messages
...
> Since we are not all native speakers expecting everyone to realize that
> difference - which is completely right - may be a little optimistic,
> moreover considering that parts of those guidelines may even be written
> by non natives.
> 
> Let's say I interpret all "should" instances in that guideline as rules
> that don't need to be strictly enforced, I see that the Change-Id
> "should not be changed when rebasing" - this one would certainly be fun
> to watch if we didn't follow it - the blueprint "should give the name of
> a Launchpad blueprint" - I don't know any core that would not -1 a patch
> if he notices the BP reference missing - and machine targeted metadata
> "should all be grouped together at the end of the commit message" - this
> one everyone follows instinctively, so no problem.
> 
> And if we look at the i18n guidelines, almost everything is using
> should, but on reviews these are treated as strict *must* because of the
> implications.
> 
> Anyway, it's a matter of opinion and afaik in Cinder we don't even have
> a real problem with downvoting for the commit message length, I don't
> see more than 1 every couple of months or so.

Other communities have solved this by explicit reference to a standard 
defining terms like "must" and "should".

Regards,
Mike


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [magnum]swarm + compose = k8s?

2015-09-28 Thread Mike Spreitzer
> From: 王华 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: 09/28/2015 11:34 PM
> Subject: [openstack-dev] [magnum]swarm + compose = k8s?
> 
> Hi folks,
> 
> Magnum now exposes service, pod, etc to users in kubernetes coe, but
> exposes container in swarm coe. As I know, swarm is only a scheduler
> of container, which is like nova in openstack. Docker compose is a 
> orchestration program which is like heat in openstack. k8s is the 
> combination of scheduler and orchestration. So I think it is better 
> to expose the apis in compose to users which are at the same level as 
k8s.
> 

Why should the users be deprived of direct access to the Swarm API when it 
is there?

Note also that Compose addresses more general, and differently focused, 
orchestration than Kubernetes; the latter only offers homogenous scaling 
groups --- which a docker-compose.yaml file can not even describe.

Regards,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Compute API (Was Re: [nova][cinder] how to handle AZ bug 1496235?)

2015-09-28 Thread Mike Spreitzer
> From: Monty Taylor 
> To: Sylvain Bauza , "OpenStack Development 
> Mailing List (not for usage questions)" 

> Date: 09/28/2015 09:54 AM
> Subject: Re: [openstack-dev] Compute API (Was Re: [nova][cinder] how
> to handle AZ bug 1496235?)
>
> ...
> Specifically, I want "nova boot" to get me a VM with an IP address. I 
> don't want it to do fancy orchestration - I want it to not need fancy 
> orchestration, because needing fancy orchestration to get a VM  on a 
> network is not a feature.
> 
> I also VERY MUCH do not want to need Heat to get a VM. I want to use 
> Heat to do something complex. Getting a VM is not complex. It should not 

> be complex. What it's complex and to the level of needing Heat, we've 
> failed somewhere else.
> 
> Also, people should stop deploying clouds that require people to use 
> floating IPs to get basic internet access. It's a misuse of the 
construct.
> 
> Public Network "ext-net" -> shared / directly attachable
> Per-tenant Network "private" -> private network, not shared, not 
routable
> 
> If the user chooses, a router can be added with gateway set to ext-net.
> 
> This way:
> 
> nova boot --network=ext-net  -> vm dhcp'd on the public network
> nova boot --network=private  -> vm dhcp'd on the private network
> nova floating-ip-attach  -> vm gets a floating ip attached to their 
> vm from the ext-net network
> 
> All of the use cases are handled, basic things are easy (boot a vm on 
> the network works in one step) and for the 5% of cases where a floating 
> IP is actually needed (a long-lived service on a single vm that wants to 

> keep the IP and not just a DNS name across VM migrations and isn't using 

> a load-balancer) can use that.
> 
> This is, btw, the most common public cloud deployment model.
> 
> Let's stop making things harder than they need to be and serve our 
users.

As an operator, +1

Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [devstack] Is there a way to configure devstack for one flat external network using Kilo, Neutron?

2015-09-28 Thread Mike Spreitzer
Is there a way to configure devstack to install Neutron such that there is 
just one network and that is an external network and Nova can create 
Compute Instances on it, using projects of Kilo vintage?

Thanks,
Mike



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model

2015-09-15 Thread Mike Spreitzer
Monty Taylor  wrote on 09/15/2015 11:04:07 AM:

> a) an update to python-novaclient to allow a named network to be passed 
> to satisfy the "you have more than one network" - the nics argument is 
> still useful for more complex things

I am not using the latest, but rather Juno.  I find that in many places 
the Neutron CLI insists on a UUID when a name could be used.  Three cheers 
for any campaign to fix that.

And, yeah, creating VMs on a shared public network is good too.

Thanks,
mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model

2015-09-15 Thread Mike Spreitzer
"Armando M."  wrote on 09/15/2015 03:50:24 PM:

> On 15 September 2015 at 10:02, Doug Hellmann  
wrote:
> Excerpts from Armando M.'s message of 2015-09-15 09:30:35 -0700:
...
> As with the Glance image upload API discussion, this is an example
> of an extremely common use case that is either complex for the end
> user or for which they have to know something about the deployment
> in order to do it at all. The usability of an OpenStack cloud running
> neutron would be enhanced greatly if there was a simple, clear, way
> for the user to get a new VM with a public IP on any cloud without
> multiple steps on their part. 

<>

...
> 
> So this boils down to: in light of the possible ways of providing VM
> connectivity, how can we make a choice on the user's behalf? Can we 
> assume that he/she always want a publicly facing VM connected to 
> Internet? The answer is 'no'.

While it may be true that in some deployments there is no good way for the 
code to choose, I think that is not the end of the story here.  The 
motivation to do this is that in *some* deployments there *is* a good way 
for the code to figure out what to do.

Regards,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model

2015-09-15 Thread Mike Spreitzer
Clark Boylan  wrote on 09/15/2015 04:06:26 PM:

> > I also strongly recommend to users to put vms on a private network and
> > use floating ip's/load balancers. For many reasons. Such as, if you
> > don't, the ip that gets assigned to the vm helps it become a pet. you
> > can't replace the vm and get the same IP. Floating IP's and load
> > balancers can help prevent pets. It also prevents security issues with
> > DNS and IP's. Also, for every floating ip/lb I have, I usually have 3x 
or
> > more the number of instances that are on the private network. Sure its
> > easy to put everything on the public network, but it provides much 
better
> > security if you only put what you must on the public network. Consider
> > the internet. would you want to expose every device in your house
> > directly on the internet? No. you put them in a private network and 
poke
> > holes just for the stuff that does. we should be encouraging good
> > security practices. If we encourage bad ones, then it will bite us 
later
> > when OpenStack gets a reputation for being associated with 
compromises.
> There are a few issues with this. Neutron IPv6 does not support floating
> IPs. So now you have to use two completely different concepts for
> networking on a single dual stacked VM. IPv4 goes on a private network
> and you attach a floating IP. IPv6 is publicly routable. If security and
> DNS and not making pets were really the driving force behind floating
> IPs we would see IPv6 support them too. These aren't the reasons
> floating IPs exist, they exist because we are running out of IPv4
> addresses and NAT is everyones preferred solution to that problem. But
> that doesn't make it a good default for a cloud; use them if you are
> affected by an IP shortage.
> 
> Nothing prevents you from load balancing against public IPs to address
> the DNS and firewall rule concerns (basically don't make pets). This
> works great and is how OpenStack's git mirrors work.
> 
> It is also easy to firewall public IPs using Neutron via security groups
> (and possibly the firewall service? I have never used it and don't
> know). All this to say I think it is reasonable to use public shared
> networks by default particularly since IPv6 does not have any concept of
> a floating IP in Neutron so using them is just odd unless you really
> really need them and you aren't actually any less secure.

I'm really glad to see the IPv6 front opened.

But I have to say that the analysis of options for securing public 
addresses omits one case that I think is important: using an external (to 
Neutron) "appliance".  In my environment this is more or less required. 
This reinforces the bifurcation of addresses that was mentioned: some VMs 
are private and do not need any service from the external appliance, while 
others have addresses that need the external appliance on the 
public/private path.

In fact, for this reason, I have taken to using two "external" networks 
(from Neutron's point of view) --- one whose addresses are handled by the 
external appliance and one whose addresses are not.  In fact, both ranges 
of address are on the same VLAN.  This is FYI, some people have wondered 
why these thins might be done.

> Not to get too off topic, but I would love it if all the devices in my
> home were publicly routable. I can use my firewall to punch holes for
> them, NAT is not required. Unfortunately I still have issues with IPv6
> at home. Maybe one day this will be a reality :)

Frankly, given the propensity for bugs to be discovered, I am glad that 
nothing in my home is accessible from the outside (aside from the device 
that does firewall, and I worry about that too).  Not that this is really 
germane to what we want to do for internet-accessible 
applications/services.

Regards,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] [fwaas] -IPv6 support in Kilo

2015-06-02 Thread Mike Spreitzer
 From: Rukhsana Ansari rukhsana.ans...@oneconvergence.com
 To: openstack-dev@lists.openstack.org
 Date: 06/02/2015 01:59 PM
 Subject: [openstack-dev]  [neutron] [fwaas] -IPv6 support in Kilo
 
 Hi,
 
 I was browsing the code to understand IPv6  support For FWaaS in Kilo.
 
 I don't see a restriction in the db code or in reference fwaas_plugin.py
 
 However, from  this:
 https://github.com/openstack/neutron-fwaas/blob/stable/kilo/
 neutron_fwaas/services/firewall/drivers/vyatta/vyatta_fwaas.py#L126
 
 I gather that at least Vyatta does not have IPv6 firewall support.
 
 Would greatly appreciate it if someone could explain  the reasons 
 for this restriction.
 
 Thanks
 -Rukhsana

Indeed, this is a surprise to me.  
http://www.brocade.com/downloads/documents/html_product_manuals/vyatta/vyatta_5400_manual/wwhelp/wwhimpl/js/html/wwhelp.htm#href=Firewall/Firewall_Overview.02.11.html#1749726
 
indicates that the Vyatta 5400, at least, definitely has firewall 
functionality.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] IPv4 transition/interoperation with IPv6

2015-05-06 Thread Mike Spreitzer
Robert Li (baoli) ba...@cisco.com wrote on 05/05/2015 09:02:08 AM:

 Currently dual stack is supported. Can you be specific on what 
 interoperation/transition techniques you are interested in? We’ve 
 been thinking about NAT64 (stateless or stateful).
 
 thanks,
 Robert
 
 On 5/4/15, 9:56 PM, Mike Spreitzer mspre...@us.ibm.com wrote:
 
 Does Neutron support any of the 4/6 interoperation/transition 
 techniques?  I wear an operator's hat nowadays, and want to make 
 IPv6 as useful and easy to use as possible for my tenants.  I think 
 the interoperation/transition techniques will play a big role in this. 


Is dual stacking working in routers now?  At the moment I am still using 
Juno, but plan to move to Kilo.

I want to encourage my tenants to use as much IPv6 as possible.  But I 
expect some will have to keep some of their workload on v4 (I know there 
is on-going work to get many application frameworks up to v6 speed, and it 
is not complete yet).  I expect some tenants could be mixed: some workload 
on v4 and some on v6.  Such a tenant would appreciate a NAT between his v6 
space and his v4 space.  This is the easiest cases --- sections 2.5 and 
2.6 --- of RFC 6144.

I would prefer to do it in a stateless way if possible.  That would be 
pretty easy if Neutron and Nova were willing to accept an IPv6 subnet that 
is much smaller than 2^64 addresses.  I see that my macs differ only in 
their last 24 bits.

Some tenants could put their entire workload on v6, but that workload 
would be unreachable from customers of all those ISPs (such as mine, 
CableVision) that deny IPv6 service to their customers.  There are 
techniques for coping, and Teredo looks like a pretty good one.  It has 
been shipped in Windows for years.  Yet I can not find a Windows machine 
where the Teredo actually works.  What's up with that?  If Windows somehow 
got its Teredo, or other, act together, that would be only half the job; 
Teredo requires something from the server side as well, right?

Supposing a focus on mobile, where IPv6 is much more available, and/or 
progress by Microsoft and/or other ISPs, my tenant might face a situation 
where his clients could come in over v6 but some of his servers still have 
to run on v4.  That's section 2.3 of RFC 6144.

While I am a Neutron operator, I am also a customer of a lower layer 
network provider.  That network provider will happily give me a few /64. 
How do I serve IPv6 subnets to lots of my tenants?  In the bad old v4 days 
this would be easy: a tenant puts all his stuff on his private networks 
and NATs (e.g., floating IP) his edge servers onto a public network --- no 
need to align tenant private subnets with public subnets.  But with no NAT 
for v6, there is no public/private distinction --- I can only give out the 
public v6 subnets that I am given.  Yes, NAT is bad.  But not being able 
to get your job done is worse.


Sean M. Collins s...@coreitpro.com wrote on 05/05/2015 06:26:28 AM:

 I think that Neutron exposes enough primitives through the API that
 advanced services for handling your transition technique of choice could
 be built.

I think that is right, if I am willing to assume Neutron is using OVS --- 
or build a bunch of alternatives that correspond to all the Neutron 
plugins and mechanisms that I might encounter.  And it would feel a lot 
like Neutron implementation work.  Really, it is one instance of doing 
some NFV.


Thanks,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [neutron] IPv4 transition/interoperation with IPv6

2015-05-04 Thread Mike Spreitzer
Does Neutron support any of the 4/6 interoperation/transition techniques? 
I wear an operator's hat nowadays, and want to make IPv6 as useful and 
easy to use as possible for my tenants.  I think the 
interoperation/transition techniques will play a big role in this.

Thanks,
Mike__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [neutron] multiple external networks on the same host NIC

2015-04-25 Thread Mike Spreitzer
As an operator, is there a way I can create two external networks from 
Neutron's point of view, where both of those networks are accessed through 
the same host NIC?  Obviously those networks would be using different 
subnets.  I need this sort of thing because the two subnets are treated 
differently by the stuff outside of OpenStack, so I need a way that a 
tenant can get a floating IP of the sort he wants.  Since Neutron equates 
floating IP allocation pools with external networks, I need two external 
networks.

I asked this on openstack-operators, and got one response saying my 
choices are either to use VLAN tagging on that NIC or manually make more 
OVS switches that ultimately get their external connectivity through the 
same NIC.  The VLAN tagging is problematic in my environment.  Is there 
any other choice?

BTW, I am currently using Juno but plan to move to Kilo.

BTW, is Neutron creating any ebtables rules that will get in my way?

Thanks,
Mike__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] Neutron scaling datapoints?

2015-04-08 Thread Mike Spreitzer
Are you looking at scaling the numbers of tenants, Neutron routers, and 
tenant networks as you scale hosts and guests?  I think this is a 
plausible way to grow.  The compartmentalizations that comes with growing 
those things may make a difference in results.

Thanks,
Mike



From:   Neil Jerram neil.jer...@metaswitch.com
To: openstack-dev@lists.openstack.org
Date:   04/08/2015 12:29 PM
Subject:[openstack-dev] [neutron] Neutron scaling datapoints?



My team is working on experiments looking at how far the Neutron server
will scale, with increasing numbers of compute hosts and VMs.  Does
anyone have any datapoints on this that they can share?  Or any clever
hints?

I'm already aware of the following ones:

https://javacruft.wordpress.com/2014/06/18/168k-instances/
 Icehouse
 118 compute hosts
 80 Neutron server processes (10 per core on each of 8 cores, on the
 controller node)
 27,000 VMs - but only after disabling all security/iptables

http://www.opencontrail.org/openstack-neutron-at-scale/
 1000 hosts
 5000 VMs
 3 Neutron servers (via a load balancer)
 But doesn't describe if any specific configuration is needed for this.
 (Other than using OpenContrail! :-))

Many thanks!
 Neil

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Heat: autoscaling across availability zones

2015-02-28 Thread Mike Spreitzer
 From: Weidong Shao weidongs...@gmail.com
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Date: 02/28/2015 02:12 PM
 Subject: [openstack-dev] Heat: autoscaling across availability zones
 
 From the  heat template reference doc, it seems that auto-scaling 
 across AZs is not supported.   Is there any blueprint or plan in 
 adding this support?

Yes.  Here are some ways you can find that work:

https://blueprints.launchpad.net/openstack/?searchtext=availabilityzone

https://review.openstack.org/#/q/message:availability+message:zone,n,z 
includes some relevant answers; I have not yet figured out how to narrow 
the query to match only items in the heat and heat-specs projects.

Subscribe to notifications from Gerrit about activity in the Heat project, 
watch the email subject lines as they whizz by.

Regards,
Mike__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] AutoScalingGroup with LB Member

2014-11-03 Thread Mike Spreitzer
Cameron Seader csea...@suse.com wrote on 11/03/2014 06:22:32 PM:

 Please if you can help me out..
 
 Here is my heat stack
 http://paste.opensuse.org/80575159
 for some reason its failing on the member address.. not sure the syntax
 is right.

Compare with 
https://github.com/openstack/heat-templates/blob/master/hot/asg_of_stacks.yaml

Perhaps the text in the Template Guide (
http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::AutoScalingGroup
) is confusing on this point.  The value of the resource property should 
be the definition of one resource (i.e., the value to which the resource 
name is mapped in the YAML), not a set of resource definitions.

Regards,
Mike


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat][ceilometer]: scale up/ down based on number of instances in a group

2014-10-28 Thread Mike Spreitzer
Daniel Comnea comnea.d...@gmail.com wrote on 10/27/2014 07:16:32 AM:

 Yes i did but if you look at this example
 
 
https://github.com/openstack/heat-templates/blob/master/hot/autoscaling.yaml

 

 the flow is simple:

 CPU alarm in Ceilometer triggers the type: OS::Heat::ScalingPolicy
 which then triggers the type: OS::Heat::AutoScalingGroup

Actually the ScalingPolicy does not trigger the ASG.  BTW, 
ScalingPolicy is mis-named; it is not a full policy, it is only an 
action (the condition part is missing --- as you noted, that is in the 
Ceilometer alarm).  The so-called ScalingPolicy does the action itself 
when triggered.  But it respects your configured min and max size.

What are you concerned about making your scaling group smaller than your 
configured minimum?  Just checking here that there is not a 
misunderstanding.

As Clint noted, there is a large-scale effort underway to make Heat 
maintain what it creates despite deletion of the underlying resources.

There is also a small-scale effort underway to make ASGs recover from 
members stopping proper functioning for whatever reason.  See 
https://review.openstack.org/#/c/127884/ for a proposed interface and 
initial implementation.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [DevStack] Proposal - add support for Markdown for docs

2014-10-27 Thread Mike Spreitzer
Sean Dague s...@dague.net wrote on 10/27/2014 09:13:27 AM:

 From: Sean Dague s...@dague.net
 
 On 10/22/2014 11:10 AM, Collins, Sean wrote:
  With some xargs, sed, and pandoc - I now present to you the first
  attempt at converting the DevStack docs to RST, and making the doc 
build
  look similar to other projects.
  
  https://review.openstack.org/130241
  
  It is extremely rough, I basically ran everything through Pandoc and
  cleaned up any errors that Sphinx spat out. I'm sure there is a lot of
  work that needs to be done to format it to be more readable - but I'm
  pretty pleased with the result.
 
 +2, you are awesome for taking this on! Thanks much.
 
-Sean

Yes, thank you very much --- both for the doc infrastructure and the docs 
that you will write using it.

Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] How can we get more feedback from users?

2014-10-24 Thread Mike Spreitzer
Angus Salkeld asalk...@mirantis.com wrote on 10/24/2014 12:32:04 AM:

 I have felt some grumblings about usability issues with Heat 
 templates/client/etc..
 and wanted a way that users could come and give us feedback easily 
 (low barrier). I started an etherpad (https://
 etherpad.openstack.org/p/heat-useablity-improvements) - the first 
 win is it is spelt wrong :-O

 We now have some great feedback there in a very short time, most of 
 this we should be able to solve.

 This lead me to think, should OpenStack have a more general 
 mechanism for users to provide feedback. The idea is this is not 
 for bugs or support, but for users to express pain points, requests 
 for features and docs/howtos.

 It's not easy to improve your software unless you are listening to your 
users.

 Ideas?

I very much agree with this.

I am actually surprised that OpenStack does not have something fairly 
formal and organized about this.  I suppose it is part of the TC's job, 
but I think we need more than they can do.  I would suggest some sort of 
user's council that gets involved in blueprint and change reviews. Perhaps 
after first working toward some degree of consensus and some degree of 
shaping what the developers work on in each release (this latter is part 
of the overall program of improving review queue speed, by better focusing 
developers and reviewers on some shared agenda).

Other products have discussion fora, some explicitly dedicated to 
feedback.  You could approximate this with etherpads, or use some real 
discussion forum platform.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] add cyclomatic complexity check to pep8 target

2014-10-16 Thread Mike Spreitzer
I like the idea of measuring complexity.  I looked briefly at `python -m 
mccabe`.  It seems to measure each method independently.  Is this really 
fair?  If I have a class with some big methods, and I break it down into 
more numerous and smaller methods, then the largest method gets smaller, 
but the number of methods gets larger.  A large number of methods is 
itself a form of complexity.  It is not clear to me that said re-org has 
necessarily made the class easier to understand.  I can also break one 
class into two, but it is not clear to me that the project has necessarily 
become easier to understand.  While it is true that when you truly make a 
project easier to understand you sometimes break it into more classes, it 
is also true that you can do a bad job of re-organizing a set of classes 
while still reducing the size of the largest method.  Has the McCabe 
metric been evaluated on Python projects?  There is a danger in focusing 
on what is easy to measure if that is not really what you want to 
optimize.

BTW, I find that one of the complexity issues for me when I am learning 
about a Python class is doing the whole-program type inference so that I 
know what the arguments are.  It seems to me that if you want to measure 
complexity of Python code then something like the complexity of the 
argument typing should be taken into account.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [ceilometer] [heat] grace period for an alarm?

2014-10-14 Thread Mike Spreitzer
Can a Ceilometer client create an alarm with a grace period?  That is, an 
initial period of time during which alarms are suppressed.  For a related 
concept, see the grace period in an AWS autoscaling group.  Note that in 
an OpenStack heat template, the author can not write an arithmetic 
expression --- any time written must be a literal (and thus apply equally 
to all members of a scaling group, regardless of when they were created).

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Openstack-dev][nova] git fetch fails with permission denied (publickey)

2014-10-06 Thread Mike Spreitzer
Jennifer Mulsow/Austin/IBM@IBMUS wrote on 10/06/2014 06:45:21 PM:

 I have followed the instructions on https://wiki.openstack.org/wiki/
 Gerrit_Workflow and am trying to fetch the nova repository, but it 
 fails with Permission denied (publickey). I believe this has 
 something to do with my specific account on review.openstack.org, 
 but not sure what.
 
 git remote add origin ssh://jmul...@review.openstack.org:29418/
 openstack/nova.git
 git fetch origin
 Permission denied (publickey).
 fatal: Could not read from remote repository.
 
 Please make sure you have the correct access rights
 and the repository exists.

Maybe I am blind tonight, but I do not see where the Gerrit_Workflow wiki 
page is telling you to `git fetch origin`.  When starting from scratch 
(which I suppose you are), you want a command like this:

git clone git://git.openstack.org/openstack/nova.git

Your SSH test looks valid to me, I do not see why it should fail.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO] a need to assert user ownership in preserved state

2014-10-01 Thread Mike Spreitzer
Clint Byrum cl...@fewbar.com wrote on 10/01/2014 09:50:33 PM:

 Recently we've been testing image based updates using TripleO, and we've
 run into an interesting conundrum.
 
 Currently, our image build scripts create a user per service for the
 image. We don't, at this time, assert a UID, so it could get any UID in
 the /etc/passwd database of the image.
 
 However, if we add a service that happens to have its users created
 before a previously existing service, the UID's shift by one. When
 this new image is deployed, the username might be 'ceilometer', but
 /mnt/state/var/lib/ceilometer is now owned by 'cinder'.

I do not understand the problem statement. Unfortunately, I am not 
familiar with image based updates using TripleO.  What is updating what? 
If the UIDs are not asserted, what UIDs shift by one?  Is this because 
some files keep owner UID while the some UID=name binding in /etc/passwd 
changes? Or the other way around?  Why would there be a change in either?

If there is no short answer, don't worry about it.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker

2014-09-24 Thread Mike Spreitzer
Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM:

 On 09/24/2014 03:31 PM, Alan Kavanagh wrote:
 Steven
 I have to ask what is the motivation and benefits we get from 
 integrating Kubernetes into Openstack? Would be really useful if you
 can elaborate and outline some use cases and benefits Openstack and 
 Kubernetes can gain. 
  
 /Alan
  
 Alan,
 
 I am either unaware or ignorant of another Docker scheduler that is 
 currently available that has a big (100+ folks) development 
 community.  Kubernetes meets these requirements and is my main 
 motivation for using it to schedule Docker containers.  There are 
 other ways to skin this cat - The TripleO folks wanted at one point 
 to deploy nova with the nova docker VM manager to do such a thing.  
 This model seemed a little clunky to me since it isn't purpose built
 around containers.

Does TripleO require container functionality that is not available
when using the Docker driver for Nova?

As far as I can tell, the quantitative handling of capacities and
demands in Kubernetes is much inferior to what Nova does today.

 As far as use cases go, the main use case is to run a specific 
 Docker container on a specific Kubernetes minion bare metal host.

If TripleO already knows it wants to run a specific Docker image
on a specific host then TripleO does not need a scheduler.

 These docker containers are then composed of the various config 
 tools and services for each detailed service in OpenStack.  For 
 example, mysql would be a container, and tools to configure the 
 mysql service would exist in the container.  Kubernetes would pass 
 config options for the mysql database prior to scheduling

I am not sure what is meant here by pass config options nor how it
would be done prior to scheduling; can you please clarify?
I do not imagine Kubernetes would *choose* the config values,
K8s does not know anything about configuring OpenStack.
Before scheduling, there is no running container to pass
anything to.

   and once 
 scheduled, Kubernetes would be responsible for connecting the 
 various containers together.

Kubernetes has a limited role in connecting containers together.
K8s creates the networking environment in which the containers
*can* communicate, and passes environment variables into containers
telling them from what protocol://host:port/ to import each imported
endpoint.  Kubernetes creates a universal reverse proxy on each
minion, to provide endpoints that do not vary as the servers
move around.
It is up to stuff outside Kubernetes to decide
what should be connected to what, and it is up to the containers
to read the environment variables and actually connect.

Regards,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker

2014-09-24 Thread Mike Spreitzer
Clint Byrum cl...@fewbar.com wrote on 09/25/2014 12:13:53 AM:

 Excerpts from Mike Spreitzer's message of 2014-09-24 20:49:20 -0700:
  Steven Dake sd...@redhat.com wrote on 09/24/2014 11:02:49 PM:
   ...
  ...
  Does TripleO require container functionality that is not available
  when using the Docker driver for Nova?
  
  As far as I can tell, the quantitative handling of capacities and
  demands in Kubernetes is much inferior to what Nova does today.
  
 
 Yes, TripleO needs to manage baremetal and containers from a single
 host. Nova and Neutron do not offer this as a feature unfortunately.

In what sense would Kubernetes manage baremetal (at all)?
By from a single host do you mean that a client on one host
can manage remote baremetal and containers?

I can see that Kubernetes allows a client on one host to get
containers placed remotely --- but so does the Docker driver for Nova.

 
   As far as use cases go, the main use case is to run a specific 
   Docker container on a specific Kubernetes minion bare metal host.

Clint, in another branch of this email tree you referred to
the VMs that host Kubernetes.  How does that square with
Steve's text that seems to imply bare metal minions?

I can see that some people have had much more detailed design
discussions than I have yet found.  Perhaps it would be helpful
to share an organized presentation of the design thoughts in
more detail.

  
  If TripleO already knows it wants to run a specific Docker image
  on a specific host then TripleO does not need a scheduler.
  
 
 TripleO does not ever specify destination host, because Nova does not
 allow that, nor should it. It does want to isolate failure domains so
 that all three Galera nodes aren't on the same PDU, but we've not really
 gotten to the point where we can do that yet.

So I am still not clear on what Steve is trying to say is the main use 
case.
Kubernetes is even farther from balancing among PDUs than Nova is.
At least Nova has a framework in which this issue can be posed and solved.
I mean a framework that actually can carry the necessary information.
The Kubernetes scheduler interface is extremely impoverished in the
information it passes and it uses GO structs --- which, like C structs,
can not be subclassed.
Nova's filter scheduler includes a fatal bug that bites when balancing and 
you want more than
one element per area, see https://bugs.launchpad.net/nova/+bug/1373478.
However: (a) you might not need more than one element per area and
(b) fixing that bug is a much smaller job than expanding the mind of K8s.

Thanks,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker

2014-09-23 Thread Mike Spreitzer
I don't know if anyone else has noticed, but you can not install Cinder 
inside a container.  Cinder requires an iSCSI package that fails to 
install; its install script tries to launch the daemon, and that fails.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] naming of provider template for docs

2014-09-19 Thread Mike Spreitzer
On further thought, I noticed that template-based resource also 
describes an AWS::CloudFormation::Stack; and since those are 
template-based, you could well describe them as custom too.

Would you consider nested stack to also describe resources of other 
types that are implemented by Python code (e.g., for scaling groups) that 
creates another stack?

I think the name we are trying to settle upon means nested stack that is 
created by supplying, as the resource type seen after mapping through the 
effective environment, a URL reference to a template.  I really think 
there is no hope of coming up with a reasonable name that includes all of 
that idea.  And in my own work, I have not needed a name that has that 
specific meaning.  I find myself more interested in nested stacks, 
regardless of the surface details of how they are requested.  Yes, the 
surface details have to be handled in some cases (e.g., balancing a 
scaling group across AZs), but that is a relatively small and confined 
matter --- at least in my own work; YMMV.  So my new bottom line is this: 
(1) for the name of the surface syntax details pick any short name that is 
not too confusing/awful, (2) in the documentation closely bind that name 
to a explanation that lays out all the critical parts of the idea, and (3) 
everywhere that you care about nested stacks regardless of the surface 
details of how they are requested in a template, use the term nested 
stack.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] naming of provider template for docs

2014-09-18 Thread Mike Spreitzer
Angus Salkeld asalk...@mirantis.com wrote on 09/18/2014 09:33:56 PM:

 Hi

 I am trying to add some docs to openstack-manuals hot_guide about
 using provider templates : https://review.openstack.org/#/c/121741/

 Mike has suggested we use a different term, he thinks provider is 
 confusing. 
 I agree that at the minimum, it is not very descriptive.

 Mike has suggested nested stack, I personally think this means 
something a 
 bit more general to many of us (it includes the concept of aws stacks) 
and may
 I suggest template resource - note this is even the class name for
 this exact functionality.
 
 Thoughts?

 Option 1) stay as is provider templates
 Option 2) nested stack
 Option 3) template resource

Thanks for rising to the documentation challenge and trying to get good 
terminology.

I think your intent is to describe a category of resources, so your option 
3 is superior to option 1 --- the thing being described is not a template, 
it is a resource (made from a template).

I think

Option 4) custom resource

would be even better.  My problem with template resource is that, to 
someone who does not already know what it means, this looks like it might 
be a kind of resource that is a template (e.g., for consumption by some 
other resource that does something with a template), rather than itself 
being something made from a template.  If you want to follow this 
direction to something perfectly clear, you might try templated resource 
(which is a little better) or template-based resource (which I think is 
pretty clear but a bit wordy) --- but an AWS::CloudFormation::Stack is 
also based on a template.  I think that if you try for a name that really 
says all of the critical parts of the idea, you will get something that is 
too wordy and/or awkward.  It is true that custom resource begs the 
question of how the user accomplishes her customization, but at least now 
we have the reader asking the right question instead of being misled.

I agree that nested stack is a more general concept.  It describes the 
net effect, which the things we are naming have in common with 
AWS::CloudFormation::Stack.  I think it would make sense for our 
documentation to say something like both an AWS::CloudFormation::Stack 
and a custom resource are ways to specify a nested stack.

Thanks,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [heat] Confused about the future of health maintenance and OS::Heat::HARestarter

2014-09-17 Thread Mike Spreitzer
Background: Health maintenance is very important to users, and I have 
users who want to do it now and into the future.  Today a Heat user can 
write a template that maintains the health of a resource R.  The detection 
of a health problem can be done by anything that hits a webhook.  That 
generality is important; it is not sufficient to determine health by 
looking at what physical and/or virtual resources exist, it is also highly 
desirable to test whether these things are functioning well (e.g., the URL 
based health checking possible through an OS::Neutron::Pool; e.g., the 
user has his own external system that detects health problems).  The 
webhook is provided by an OS::Heat::HARestarter (note the name bug: such a 
thing does not restart anything, rather it deletes and re-creates a given 
resource and all its dependents) that deletes and re-creates R and its 
health detection/recovery wiring.  For a more specific example, consider 
the case of detection using the services of an OS::Neutron::Pool.  Note 
that it is not even necessary for there to be workload traffic through the 
associated OS::Neutron::LoadBalancer; all we are using here is the 
monitoring prescribed by the Pool's OS::Neutron::HealthMonitor.  The 
user's template has, in addition to R, three things: (1) an 
OS::Neutron::PoolMember that puts R in the Pool, (2) an 
OS::Heat::HARestarter that deletes and re-creates R and all its 
dependents, and (3) a Ceilometer alarm that detects when Neutron is 
reporting that the PoolMember is unhealthy and responds by hitting the 
HARestarter's webhook.  Note that all three of those are dependent on R, 
and thus are deleted and re-created when the HARestarter's webhook is hit; 
this avoids most of the noted issues with HARestarter.  R can be a stack 
that includes both a Nova server and an OS::Neutron::Port, to work around 
a Nova bug with implicit ports.

There is a movement afoot to remove HARestarter.  My concern is what can 
users do, now and into the future.  The first and most basic issue is 
this: at every step in the roadmap, it must be possible for users to 
accomplish health maintenance.  The second issue is easing the impact on 
what users write.  It would be pretty bad if the roadmap looks like this: 
before point X, users can only accomplish health maintenance as I outlined 
above, and from point X onward the user has to do something different. 
That is, there should be a transition period during which users can do 
things either the old way or the new way.  It would be even better if we, 
or a cloud provider, could provide an abstraction that will be usable 
throughout the roadmap (once that abstraction becomes available).  For 
example, if there were a resource type OS::Heat::ReliableAutoScalingGroup 
that adds health maintenance functionality (with detection by an 
OS::Neutron::Pool and exposure of per-member webhooks usable by anything) 
to OS::Heat::AutoScalingGroup.  Once some other way to do that maintenance 
becomes available, the implementation of 
OS::Heat::ReliableAutoScalingGroup could switch to that without requiring 
any changes to users' templates.  If at some point in the future 
OS::Heat::ReliableAutoScalingGroup becomes exactly equivalent to 
OS::Heat::AutoScalingGroup then we could deprecate 
OS::Heat::ReliableAutoScalingGroup and, at a later time, remove it.  Even 
better: since health maintenance is not logically connected to scaling 
group membership, make the abstraction be simply OS::Heat::HealthyResource 
(i.e., it is about a single resource regardless of whether it is a member 
of a scaling group) rather than OS::Heat::ReliableAutoScalingGroup. 
Question: would that abstraction (including the higher level detection and 
exposure of re-creation webhook) be implementable (or a no-op) in the 
planned future?

To aid in understanding: while it may be distasteful for a resource like 
HARestarter to tweak its containing stack, the critical question is 
whether it will remain *possible* throughout a transition period.  Is 
there an issue with such hacks being *possible* throughout a reasonable 
transition period?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] referencing the index of a ResourceGroup

2014-09-11 Thread Mike Spreitzer
Steven Hardy sha...@redhat.com wrote on 09/11/2014 04:21:18 AM:

 On Wed, Sep 10, 2014 at 04:44:01PM -0500, Jason Greathouse wrote:
 I'm trying to find a way to create a set of servers and attach a 
new
 volume to each server. 
 ...
 
 Basically creating lots of resource groups for related things is the 
wrong
 pattern.  You need to create one nested stack template containing the
 related things (Server, Volume and VolumeAttachment in this case), and 
use
 ResourceGroup to multiply them as a unit.
 
 I answered a similar question here on the openstack general ML recently
 (which for future reference may be a better ML for usage questions like
 this, as it's not really development discussion):
 
 
http://lists.openstack.org/pipermail/openstack/2014-September/009216.html
 
 Here's another example which I used in a summit demo, which I think
 basically does what you need?
 
 https://github.com/hardys/demo_templates/tree/master/
 juno_summit_intro_to_heat/example3_server_with_volume_group

There is also an example of exactly this under review.  See 
https://review.openstack.org/#/c/97366/

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [heat] Re: Change in openstack/heat[master]: Implement AZ spanning for ASGs

2014-08-27 Thread Mike Spreitzer
You offered to share ideas about a different way to approach spanning AZs 
for OS::Heat::AutoScalingGroup.  I am interested.  Can we discuss it here?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [heat] heat.conf.sample is not up to date

2014-08-24 Thread Mike Spreitzer
What is going on with this?  If I do a fresh clone of heat and run `tox 
-epep8` then I get that complaint.  If I then run the recommended command 
to fix it, and then `tox -epep8` again, I get the same complaint again --- 
and with different differences exhibited!  The email below carries a 
typescript showing this.

What I really need to know is what to do when committing a change that 
really does require a change in the sample configuration file.  Of course 
I tried running generate_sample.sh, but `tox -epep8` still complains. What 
is the right procedure to get a correct sample committed?  BTW, I am doing 
the following admittedly risky thing: I run DevStack, and make my changes 
in /opt/stack/heat/.

Thanks,
Mike

- Forwarded by Mike Spreitzer/Watson/IBM on 08/24/2014 03:03 AM -

From:   ubuntu@mjs-dstk-821a (Ubuntu)
To: Mike Spreitzer/Watson/IBM@IBMUS, 
Date:   08/24/2014 02:55 AM
Subject:fresh flake fail



ubuntu@mjs-dstk-821a:~/code$ git clone 
git://git.openstack.org/openstack/heat.git
Cloning into 'heat'...
remote: Counting objects: 49690, done.
remote: Compressing objects: 100% (19765/19765), done.
remote: Total 49690 (delta 36660), reused 39014 (delta 26526)
Receiving objects: 100% (49690/49690), 7.92 MiB | 7.29 MiB/s, done.
Resolving deltas: 100% (36660/36660), done.
Checking connectivity... done.
ubuntu@mjs-dstk-821a:~/code$ cd heat
ubuntu@mjs-dstk-821a:~/code/heat$ tox -epep8
pep8 create: /home/ubuntu/code/heat/.tox/pep8
pep8 installdeps: -r/home/ubuntu/code/heat/requirements.txt, 
-r/home/ubuntu/code/heat/test-requirements.txt
pep8 develop-inst: /home/ubuntu/code/heat
pep8 runtests: PYTHONHASHSEED='0'
pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn 
bin/heat-api-cloudwatch bin/heat-engine bin/heat-manage contrib
pep8 runtests: commands[1] | 
/home/ubuntu/code/heat/tools/config/check_uptodate.sh
--- /tmp/heat.ep2CBe/heat.conf.sample2014-08-24 06:52:54.16484 +
+++ etc/heat/heat.conf.sample2014-08-24 06:48:13.66484 +
@@ -164,7 +164,7 @@
 
#allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions
 
 # Qpid broker hostname. (string value)
-#qpid_hostname=heat
+#qpid_hostname=localhost
 
 # Qpid broker port. (integer value)
 #qpid_port=5672
@@ -221,7 +221,7 @@
 
 # The RabbitMQ broker address where a single node is used.
 # (string value)
-#rabbit_host=heat
+#rabbit_host=localhost
 
 # The RabbitMQ broker port where a single node is used.
 # (integer value)
check_uptodate.sh: heat.conf.sample is not up to date.
check_uptodate.sh: Please run 
/home/ubuntu/code/heat/tools/config/generate_sample.sh.
ERROR: InvocationError: 
'/home/ubuntu/code/heat/tools/config/check_uptodate.sh'
pep8 runtests: commands[2] | 
/home/ubuntu/code/heat/tools/requirements_style_check.sh requirements.txt 
test-requirements.txt
pep8 runtests: commands[3] | bash -c find heat -type f -regex '.*\.pot?' 
-print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null
___ summary 

ERROR:   pep8: commands failed
ubuntu@mjs-dstk-821a:~/code/heat$ 
ubuntu@mjs-dstk-821a:~/code/heat$ 
ubuntu@mjs-dstk-821a:~/code/heat$ tools/config/generate_sample.sh
ubuntu@mjs-dstk-821a:~/code/heat$ 
ubuntu@mjs-dstk-821a:~/code/heat$ 
ubuntu@mjs-dstk-821a:~/code/heat$ 
ubuntu@mjs-dstk-821a:~/code/heat$ tox -epep8
pep8 develop-inst-noop: /home/ubuntu/code/heat
pep8 runtests: PYTHONHASHSEED='0'
pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn 
bin/heat-api-cloudwatch bin/heat-engine bin/heat-manage contrib
pep8 runtests: commands[1] | 
/home/ubuntu/code/heat/tools/config/check_uptodate.sh
--- /tmp/heat.DqIhK5/heat.conf.sample2014-08-24 06:54:34.62884 +
+++ etc/heat/heat.conf.sample2014-08-24 06:53:51.54084 +
@@ -159,10 +159,6 @@
 # Size of RPC connection pool. (integer value)
 #rpc_conn_pool_size=30
 
-# Modules of exceptions that are permitted to be recreated
-# upon receiving exception data from an rpc call. (list value)
-#allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions
-
 # Qpid broker hostname. (string value)
 #qpid_hostname=heat
 
@@ -301,15 +297,6 @@
 # Heartbeat time-to-live. (integer value)
 #matchmaker_heartbeat_ttl=600
 
-# Host to locate redis. (string value)
-#host=127.0.0.1
-
-# Use this port to connect to redis host. (integer value)
-#port=6379
-
-# Password for Redis server (optional). (string value)
-#password=None
-
 # Size of RPC greenthread pool. (integer value)
 #rpc_thread_pool_size=64
 
@@ -1229,6 +1216,22 @@
 #hash_algorithms=md5
 
 
+[matchmaker_redis]
+
+#
+# Options defined in oslo.messaging
+#
+
+# Host to locate redis. (string value)
+#host=127.0.0.1
+
+# Use this port to connect to redis host. (integer value)
+#port=6379
+
+# Password for Redis server (optional). (string value)
+#password=None
+
+
 [matchmaker_ring]
 
 #
check_uptodate.sh: heat.conf.sample is not up to date

Re: [openstack-dev] [heat] heat.conf.sample is not up to date

2014-08-24 Thread Mike Spreitzer
The differences do not look to me like ones that would follow from a 
difference in hash seed.

Clint Byrum cl...@fewbar.com wrote on 08/24/2014 05:00:21 AM:

 From: Clint Byrum cl...@fewbar.com
 To: openstack-dev openstack-dev@lists.openstack.org, 
 Date: 08/24/2014 05:02 AM
 Subject: Re: [openstack-dev] [heat] heat.conf.sample is not up to date
 
 Guessing this is due to the new tox feature which randomizes python's
 hash seed.
 
 Excerpts from Mike Spreitzer's message of 2014-08-24 00:10:42 -0700:
  What is going on with this?  If I do a fresh clone of heat and run 
`tox 
  -epep8` then I get that complaint.  If I then run the recommended 
command 
  to fix it, and then `tox -epep8` again, I get the same complaint again 
--- 
  and with different differences exhibited!  The email below carries a 
  typescript showing this.
  
  What I really need to know is what to do when committing a change that 

  really does require a change in the sample configuration file.  Of 
course 
  I tried running generate_sample.sh, but `tox -epep8` still complains. 
What 
  is the right procedure to get a correct sample committed?  BTW, I am 
doing 
  the following admittedly risky thing: I run DevStack, and make my 
changes 
  in /opt/stack/heat/.
  
  Thanks,
  Mike
  
  - Forwarded by Mike Spreitzer/Watson/IBM on 08/24/2014 03:03 AM 
-
  
  From:   ubuntu@mjs-dstk-821a (Ubuntu)
  To: Mike Spreitzer/Watson/IBM@IBMUS, 
  Date:   08/24/2014 02:55 AM
  Subject:fresh flake fail
  
  
  
  ubuntu@mjs-dstk-821a:~/code$ git clone 
  git://git.openstack.org/openstack/heat.git
  Cloning into 'heat'...
  remote: Counting objects: 49690, done.
  remote: Compressing objects: 100% (19765/19765), done.
  remote: Total 49690 (delta 36660), reused 39014 (delta 26526)
  Receiving objects: 100% (49690/49690), 7.92 MiB | 7.29 MiB/s, done.
  Resolving deltas: 100% (36660/36660), done.
  Checking connectivity... done.
  ubuntu@mjs-dstk-821a:~/code$ cd heat
  ubuntu@mjs-dstk-821a:~/code/heat$ tox -epep8
  pep8 create: /home/ubuntu/code/heat/.tox/pep8
  pep8 installdeps: -r/home/ubuntu/code/heat/requirements.txt, 
  -r/home/ubuntu/code/heat/test-requirements.txt
  pep8 develop-inst: /home/ubuntu/code/heat
  pep8 runtests: PYTHONHASHSEED='0'
  pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn 

  bin/heat-api-cloudwatch bin/heat-engine bin/heat-manage contrib
  pep8 runtests: commands[1] | 
  /home/ubuntu/code/heat/tools/config/check_uptodate.sh
  --- /tmp/heat.ep2CBe/heat.conf.sample2014-08-24 06:52:54.16484 
+
  +++ etc/heat/heat.conf.sample2014-08-24 06:48:13.66484 +
  @@ -164,7 +164,7 @@
  
  
 
#allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions
  
   # Qpid broker hostname. (string value)
  -#qpid_hostname=heat
  +#qpid_hostname=localhost
  
   # Qpid broker port. (integer value)
   #qpid_port=5672
  @@ -221,7 +221,7 @@
  
   # The RabbitMQ broker address where a single node is used.
   # (string value)
  -#rabbit_host=heat
  +#rabbit_host=localhost
  
   # The RabbitMQ broker port where a single node is used.
   # (integer value)
  check_uptodate.sh: heat.conf.sample is not up to date.
  check_uptodate.sh: Please run 
  /home/ubuntu/code/heat/tools/config/generate_sample.sh.
  ERROR: InvocationError: 
  '/home/ubuntu/code/heat/tools/config/check_uptodate.sh'
  pep8 runtests: commands[2] | 
  /home/ubuntu/code/heat/tools/requirements_style_check.sh 
requirements.txt 
  test-requirements.txt
  pep8 runtests: commands[3] | bash -c find heat -type f -regex 
'.*\.pot?' 
  -print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null
  ___ summary 
  
  ERROR:   pep8: commands failed
  ubuntu@mjs-dstk-821a:~/code/heat$ 
  ubuntu@mjs-dstk-821a:~/code/heat$ 
  ubuntu@mjs-dstk-821a:~/code/heat$ tools/config/generate_sample.sh
  ubuntu@mjs-dstk-821a:~/code/heat$ 
  ubuntu@mjs-dstk-821a:~/code/heat$ 
  ubuntu@mjs-dstk-821a:~/code/heat$ 
  ubuntu@mjs-dstk-821a:~/code/heat$ tox -epep8
  pep8 develop-inst-noop: /home/ubuntu/code/heat
  pep8 runtests: PYTHONHASHSEED='0'
  pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn 

  bin/heat-api-cloudwatch bin/heat-engine bin/heat-manage contrib
  pep8 runtests: commands[1] | 
  /home/ubuntu/code/heat/tools/config/check_uptodate.sh
  --- /tmp/heat.DqIhK5/heat.conf.sample2014-08-24 06:54:34.62884 
+
  +++ etc/heat/heat.conf.sample2014-08-24 06:53:51.54084 +
  @@ -159,10 +159,6 @@
   # Size of RPC connection pool. (integer value)
   #rpc_conn_pool_size=30
  
  -# Modules of exceptions that are permitted to be recreated
  -# upon receiving exception data from an rpc call. (list value)
  -
 
#allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions
  -
   # Qpid broker hostname. (string value)
   #qpid_hostname=heat
  
  @@ -301,15 +297,6 @@
   # Heartbeat time-to-live

Re: [openstack-dev] [heat] heat.conf.sample is not up to date

2014-08-24 Thread Mike Spreitzer
Morgan Fainberg morgan.fainb...@gmail.com wrote on 08/24/2014 12:01:37 
PM:

 ...
 
 Keystone saw an oddity with the new sample config generator 
 (changing how options are sorted and therefore changing the way the 
 sample config is rendered). This could be a similar / related issue. 
 
 Most of the projects stopped gating on up-to-date sample config a 
 few reasons, the first is that with external library dependencies 
 you never know when / if something upstream will break the test run 
 (e.g. New Oslo.config or new keystonemiddleware). 
 
 Now imagine that issue occurred and was blocking a gate-fixing bug 
 (happened at least a couple times). 
 
 In short, making sample config being up-to-date to merge code causes
 a lot if headaches. 
 
 Different projects handle this differently. Nova doesn't have a 
 sample config in tree, keystone updates on a semi-regular basis 
 (sometimes as part of a patch, sometimes as a separate patch). 
 Keystone team is looking at adding a simple non-voting gate job (if 
 infra doesn't mind) that will tell us the config is out of date.
 
 While it is nice to always have an updated sample config, I think it
 is not worth the breakage / issues it adds to the gate.
 
 It might make sense to standardize how we handle sample config files
 across the projects or at least standardize on removing the gate 
 block if the config is out of date. I know it was floated earlier 
 that there would be a proposal bot job (like translations) for 
 sample config files, but I don't remember the specifics of why it 
 wasn't well liked. 

Consistency would be nice.  The must-have is just to document each 
project's procedure (accurately, of course).

Thanks,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] heat.conf.sample is not up to date

2014-08-24 Thread Mike Spreitzer
Ruslan Kamaldinov rkamaldi...@mirantis.com wrote on 08/24/2014 12:30:04 
PM:

 From: Ruslan Kamaldinov rkamaldi...@mirantis.com
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org, 
 Date: 08/24/2014 12:32 PM
 Subject: Re: [openstack-dev] [heat] heat.conf.sample is not up to date
 
 On Sun, Aug 24, 2014 at 11:10 AM, Mike Spreitzer mspre...@us.ibm.com 
wrote:
  What I really need to know is what to do when committing a change that
  really does require a change in the sample configuration file.  Of 
course I
  tried running generate_sample.sh, but `tox -epep8` still complains. 
What is
  the right procedure to get a correct sample committed?  BTW, I am 
doing the
  following admittedly risky thing: I run DevStack, and make my changes 
in
  /opt/stack/heat/.
 
 Mike,
 
 It seems that you have oslo.messaging installed from master (default
 behavior of Devstack), while heat.sample.config is built for
 oslo.messaging v 1.3.1. As a quick workaround I'd suggest to downgrade
 oslo.messaging to 1.3.1 in pep8 virtual environment:
 $ cd /opt/stack/heat
 $ source .tox/pep8/bin/activate
 $ pip install oslo.messaging==1.3.1
 $ deactivate
 $ tox -e pep8 # should pass now

In that same VM in which I ran DevStack, I also made an independent clone 
of heat in ~/code/heat; my original email quoted a failure from that 
clone, hoping that it might be easier to debug (being a simpler scenario).

 Below is a typescript showing me try again there, including a trial of 
what Mathieu Gagné suggested (which has some kind of command line syntax

error, and did nothing).  You will see that I investigated and found that 
in this case tox's venv contained oslo.messaging version 1.3.1, so no 
adjustment about that was needed.  Yet it still failed.

ubuntu@mjs-dstk-821a:~/code/heat$ rm -rf .tox

ubuntu@mjs-dstk-821a:~/code/heat$ tox -evenv bash 
./tools/config/generate_sample.sh -b . -p heat -o etc/heat
usage: tox [-h] [--version] [-v] [--showconfig] [-l] [-c CONFIGFILE]
   [-e envlist] [--notest] [--sdistonly] [--installpkg PATH]
   [--develop] [--set-home] [-i URL] [-r] [--result-json PATH]
   [--hashseed SEED] [--force-dep REQ] [--sitepackages]
   [--skip-missing-interpreters]
   [args [args ...]]
tox: error: unrecognized arguments: -b . -p heat -o etc/heat

ubuntu@mjs-dstk-821a:~/code/heat$ tox -epep8
pep8 create: /home/ubuntu/code/heat/.tox/pep8
pep8 installdeps: -r/home/ubuntu/code/heat/requirements.txt, 
-r/home/ubuntu/code/heat/test-requirements.txt
pep8 develop-inst: /home/ubuntu/code/heat
pep8 runtests: PYTHONHASHSEED='0'
pep8 runtests: commands[0] | flake8 heat bin/heat-api bin/heat-api-cfn 
bin/heat-api-cloudwatch bin/heat-engine bin/heat-manage contrib
pep8 runtests: commands[1] | 
/home/ubuntu/code/heat/tools/config/check_uptodate.sh
--- /tmp/heat.UnHAZD/heat.conf.sample   2014-08-25 04:06:07.64884 
+
+++ etc/heat/heat.conf.sample   2014-08-24 06:53:51.54084 +
@@ -159,10 +159,6 @@
 # Size of RPC connection pool. (integer value)
 #rpc_conn_pool_size=30
 
-# Modules of exceptions that are permitted to be recreated
-# upon receiving exception data from an rpc call. (list value)
-#allowed_rpc_exception_modules=oslo.messaging.exceptions,nova.exception,cinder.exception,exceptions
-
 # Qpid broker hostname. (string value)
 #qpid_hostname=heat
 
@@ -301,15 +297,6 @@
 # Heartbeat time-to-live. (integer value)
 #matchmaker_heartbeat_ttl=600
 
-# Host to locate redis. (string value)
-#host=127.0.0.1
-
-# Use this port to connect to redis host. (integer value)
-#port=6379
-
-# Password for Redis server (optional). (string value)
-#password=None
-
 # Size of RPC greenthread pool. (integer value)
 #rpc_thread_pool_size=64
 
@@ -1229,6 +1216,22 @@
 #hash_algorithms=md5
 
 
+[matchmaker_redis]
+
+#
+# Options defined in oslo.messaging
+#
+
+# Host to locate redis. (string value)
+#host=127.0.0.1
+
+# Use this port to connect to redis host. (integer value)
+#port=6379
+
+# Password for Redis server (optional). (string value)
+#password=None
+
+
 [matchmaker_ring]
 
 #
check_uptodate.sh: heat.conf.sample is not up to date.
check_uptodate.sh: Please run 
/home/ubuntu/code/heat/tools/config/generate_sample.sh.
ERROR: InvocationError: 
'/home/ubuntu/code/heat/tools/config/check_uptodate.sh'
pep8 runtests: commands[2] | 
/home/ubuntu/code/heat/tools/requirements_style_check.sh requirements.txt 
test-requirements.txt
pep8 runtests: commands[3] | bash -c find heat -type f -regex '.*\.pot?' 
-print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null
_
__ summary 
_
___
ERROR:   pep8: commands failed

ubuntu@mjs-dstk-821a:~/code/heat$ . .tox/pep8/bin/activate

(pep8)ubuntu@mjs-dstk-821a:~/code/heat$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type help, copyright, credits or license for more information.
 system
Traceback (most recent call last):
  File

Re: [openstack-dev] [OpenStack][Docker] Run OpenStack Service in Docker Container

2014-08-19 Thread Mike Spreitzer
In particular, I tried to run DevStack inside an LXC a few months ago.  I 
discovered that DevStack (presumably for the sake of cinder-volume) 
pre-reqs a system package named tgt, and tgt does not succeed to install 
inside an LXC (the install script launches the daemon, but the daemon 
launch fails).

Regards,
Mike



From:   Jyoti Ranjan jran...@gmail.com
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org, 
Date:   08/18/2014 08:53 AM
Subject:Re: [openstack-dev] [OpenStack][Docker] Run OpenStack 
Service in Docker Container



I believe that everything can not go as a dock container. For e.g. 

1. compute nodes
2. baremetal provisioning
3. L3 router etc


My understanding is that container is good mechanism to deploy 
api-controller and scheduler for many services. For backend component of 
services (like nova-compute, cinder-volume if LVM is used), I that usage 
of baremetal is more appropriate (except backend component like 
cinder-volume for external devices, nova-compute proxy etc).


Just thought to check your opinion about my understanding! Need your 
views!


On Mon, Aug 18, 2014 at 3:34 PM, Jay Lau jay.lau@gmail.com wrote:
I see that there are some openstack docker images in public docker repo, 
perhaps you can check them on github to see how to use them.

[root@db03b04 ~]# docker search openstack
NAME 
DESCRIPTION STARS OFFICIAL   
AUTOMATED
ewindisch/dockenstackOpenStack development environment 
(using D...   6[OK]
jyidiego/openstack-clientAn ubuntu 12.10 LTS image that 
has nova, s...   1
dkuffner/docker-openstack-stress A docker container for openstack 
which pro...   0[OK]
garland/docker-openstack-keystone   
 
0[OK]
mpaone/openstack
 
0
nirmata/openstack-base  
 
0
balle/openstack-ipython2-client  Features Python 2.7.5, Ipython 
2.1.0 and H...   0
booleancandy/openstack_clients  
 
0[OK]
leseb/openstack-keystone
 
0
raxcloud/openstack-client   
 
0
paulczar/openstack-agent
 
0
booleancandy/openstack-clients  
 
0
jyidiego/openstack-client-rumm-ansible  
 
0
bodenr/jumpgate  SoftLayer Jumpgate WSGi OpenStack 
REST API...   0[OK]
sebasmagri/docker-marconiDocker images for the Marconi 
Message Queu...   0[OK]
chamerling/openstack-client 
 
0[OK]
centurylink/openstack-cli-wetty  This image provides a Wetty 
terminal with ...   0[OK]


2014-08-18 16:47 GMT+08:00 Philip Cheong philip.che...@elastx.se:

I think it's a very interesting test for docker. I too have been think 
about this for some time to try and dockerise OpenStack services, but as 
the usual story goes, I have plenty things I'd love to try, but there are 
only so many hours in a day...

Would definitely be interested to hear if anyone has attempted this and 
what the outcome was. 

Any suggestions on what the most appropriate service would be to begin 
with?


On 14 August 2014 14:54, Jay Lau jay.lau@gmail.com wrote:
I see a few mentions of OpenStack services themselves being containerized 
in Docker. Is this a serious trend in the community?

http://allthingsopen.com/2014/02/12/why-containers-for-openstack-services/

-- 
Thanks,

Jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Philip Cheong
Elastx | Public and Private PaaS
email: philip.che...@elastx.se
office: +46 8 557 728 10
mobile: +46 702 8170 814
twitter: @Elastx
http://elastx.se

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Thanks,

Jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack][Docker] Run OpenStack Service in Docker Container

2014-08-19 Thread Mike Spreitzer
Jay Lau jay.lau@gmail.com wrote on 08/14/2014 08:54:56 AM:
 I see a few mentions of OpenStack services themselves being 
 containerized in Docker. Is this a serious trend in the community?
 
 
http://allthingsopen.com/2014/02/12/why-containers-for-openstack-services/
 

It looks to me like the problem addressed there is managing port 
assignments.  To make progress on that you would need something that lets 
the clients know the hostport endpoints where the servers are found, and 
you would need both the clients and servers using it.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] Separating the issues around smarter/solver/joint/holistic placement

2014-08-19 Thread Mike Spreitzer
There has been a lot of discussion around these issues, let me see if I 
can break it down into pieces, hopefully in a way that allows some 
progress on one of them first.

I continue to focus on the timeless version of the problem, in which the 
placement question is simply where can we put some guest(s) now without 
making plans for whether/how things may change in the future.  There is 
also a rich discussion to have about placement over time (in other 
communities this is what scheduling means), but that is not my goal 
here.

I see essentially three issues: (i1) more sophisticated placement 
criteria, (i2) joint vs sequential decision making, and (i3) 
single-service vs cross service.  I think we could make a roadmap for 
addressing them in this order.  Do you think the issues are separable and 
attackable in this way?

Objections have been raised regarding complexity, and I would like to be 
clear about what complexity we are discussing.  For each of these three 
issues, I can think of five kinds of complexity to discuss: the 
development complexity faced by us OpenStack developers and/or a cloud 
provider who wants to add such things, the runtime costs, the API 
complexity for allowing the issue to be addressed, the development 
complexity faced by the cloud users in writing their placement problems, 
and the size of the written problems.

(i1) More sophisticated placement criteria.  For example, I think cloud 
providers would like to have a way to express a preference for 
consolidation, to help them conserve energy.  That's a fairly rich topic 
in itself, and I hope the details need not concern us here.  OTOH, cloud 
users may want to have a way to express a preference for spreading their 
guests out --- for example, spreading around within an availability zone. 
This may seem relatively boring for stateless web servers, but if you 
think about things like Zookeeper or Cassandra servers you see a serious 
preference for getting as much failure independence as you can get within 
a given AZ.  Other examples of what cloud users may want cross services, 
like placing VMs and storage volumes in close proximity (by some measure).

Let's consider a couple of concrete examples from the record.  One is 
referenced in the current revision of 
https://review.openstack.org/#/c/96543/ - namely, http://goo.gl/vuHcz2 . 
There Yathi exhibits a binary integer programming problem for placing a 
group of guests so as to minimize the sum of (host's free RAM before any 
members of this group are placed).  Minimizing that objective has a 
roughly consolidating effect in many situations, even though it does not 
directly express exactly consolidation (exactly expressing consolidation 
requires a non-linear objective).  Similarly, maximizing that objective 
has something of a tendency to spread things out, even though the 
objective does not exactly express spreading.  (BTW, I do not think we 
should assume that all hosts in an AZ have the same capacity vector.)

Another example is at 
https://wiki.openstack.org/wiki/Heat/PolicyExtension#LLMN_Anti-Collocation 
.  There we see a criterion that is a precise spreading condition, for 
spreading a group of guests to a certain precise degree.  (BTW, I tend to 
use collocation and anti-collocation when speaking of affinity stated 
in terms of location --- as opposed, for example, to network proximity.)

For any collection of more or less sophisticated placement criteria, if we 
consider the question of placing a single guest --- supposing it is 
stipulated that some guests have already had their placement determined 
and others will be determined later --- the question boils down to the 
exactly the terms that we have in Nova's FilterScheduler today: (a) which 
hosts can host this guest and (b) ranking the acceptable hosts.  Of 
course, that is not the only way to address a problem of placing several 
guests --- but it is one valid way, even if the placement problem is 
stated as a mixed integer programming problem.  Group placement problems 
are NP hard, so solving them exactly is out of the question.  The only 
question is how hard are we going to work at making a good approximation; 
this is what (i2) is about, and we will get to that below.

Let us review the two concrete examples with an eye towards the five kinds 
of complexity.  Let us start with Yathi's RAM minimization binary integer 
programming example.  There is a difficulty if we try to work that example 
in today's framework.  The objective is a function of the amount of RAM 
that was free on each host *before any group members are placed*.  Today's 
FilterScheduler works on one guest at a time, updating host states as it 
goes along; it does not offer earlier host states for use in later 
decisions.  However, today we can solve a slightly different problem --- 
which happens to be an even better approximation of a consolidation 
problem.  That is to minimize the sum of (host's available RAM just before 
the 

Re: [openstack-dev] [nova] Separating the issues around smarter/solver/joint/holistic placement

2014-08-19 Thread Mike Spreitzer
This is primarily an issue for Nova.

Mike Spreitzer/Watson/IBM@IBMUS wrote on 08/20/2014 01:21:24 AM:

 From: Mike Spreitzer/Watson/IBM@IBMUS
 To: OpenStack Development Mailing List 
openstack-dev@lists.openstack.org, 
 Date: 08/20/2014 01:24 AM
 Subject: [openstack-dev] Separating the issues around smarter/
 solver/joint/holistic placement
 
 There has been a lot of discussion around these issues, let me see 
 if I can break it down into pieces, hopefully in a way that allows 
 some progress on one of them first. 
 
 I continue to focus on the timeless version of the problem, in which
 the placement question is simply where can we put some guest(s) now 
 without making plans for whether/how things may change in the 
 future.  There is also a rich discussion to have about placement 
 over time (in other communities this is what scheduling means), 
 but that is not my goal here. 
 
 I see essentially three issues: (i1) more sophisticated placement 
 criteria, (i2) joint vs sequential decision making, and (i3) single-
 service vs cross service.  I think we could make a roadmap for 
 addressing them in this order.  Do you think the issues are 
 separable and attackable in this way? 
 
 Objections have been raised regarding complexity, and I would like
 to be clear about what complexity we are discussing.  For each of 
 these three issues, I can think of five kinds of complexity to 
 discuss: the development complexity faced by us OpenStack developers
 and/or a cloud provider who wants to add such things, the runtime 
 costs, the API complexity for allowing the issue to be addressed, 
 the development complexity faced by the cloud users in writing their
 placement problems, and the size of the written problems. 
 
 (i1) More sophisticated placement criteria.  For example, I think 
 cloud providers would like to have a way to express a preference for
 consolidation, to help them conserve energy.  That's a fairly rich 
 topic in itself, and I hope the details need not concern us here. 
 OTOH, cloud users may want to have a way to express a preference for
 spreading their guests out --- for example, spreading around within 
 an availability zone.  This may seem relatively boring for stateless
 web servers, but if you think about things like Zookeeper or 
 Cassandra servers you see a serious preference for getting as much 
 failure independence as you can get within a given AZ.  Other 
 examples of what cloud users may want cross services, like placing 
 VMs and storage volumes in close proximity (by some measure). 
 
 Let's consider a couple of concrete examples from the record.  One 
 is referenced in the current revision of https://
 review.openstack.org/#/c/96543/ - namely, http://goo.gl/vuHcz2 . 
 There Yathi exhibits a binary integer programming problem for 
 placing a group of guests so as to minimize the sum of (host's free 
 RAM before any members of this group are placed).  Minimizing that 
 objective has a roughly consolidating effect in many situations, 
 even though it does not directly express exactly consolidation 
 (exactly expressing consolidation requires a non-linear objective). 
 Similarly, maximizing that objective has something of a tendency to 
 spread things out, even though the objective does not exactly 
 express spreading.  (BTW, I do not think we should assume that all 
 hosts in an AZ have the same capacity vector.) 
 
 Another example is at https://wiki.openstack.org/wiki/Heat/
 PolicyExtension#LLMN_Anti-Collocation .  There we see a criterion 
 that is a precise spreading condition, for spreading a group of 
 guests to a certain precise degree.  (BTW, I tend to use 
 collocation and anti-collocation when speaking of affinity 
 stated in terms of location --- as opposed, for example, to network 
 proximity.) 
 
 For any collection of more or less sophisticated placement criteria,
 if we consider the question of placing a single guest --- supposing 
 it is stipulated that some guests have already had their placement 
 determined and others will be determined later --- the question 
 boils down to the exactly the terms that we have in Nova's 
 FilterScheduler today: (a) which hosts can host this guest and (b) 
 ranking the acceptable hosts.  Of course, that is not the only way 
 to address a problem of placing several guests --- but it is one 
 valid way, even if the placement problem is stated as a mixed 
 integer programming problem.  Group placement problems are NP hard, 
 so solving them exactly is out of the question.  The only question 
 is how hard are we going to work at making a good approximation; 
 this is what (i2) is about, and we will get to that below. 
 
 Let us review the two concrete examples with an eye towards the five
 kinds of complexity.  Let us start with Yathi's RAM minimization 
 binary integer programming example.  There is a difficulty if we try
 to work that example in today's framework.  The objective is a 
 function of the amount of RAM that was free

Re: [openstack-dev] OS or os are not acronyms for OpenStack

2014-08-15 Thread Mike Spreitzer
Anita Kuno ante...@anteaya.info wrote on 08/15/2014 10:38:20 AM:

 OpenStack is OpenStack. The use of openstack is also acceptable in our
 development conversations.
 
 OS or os is operating system. I am starting to see some people us OS or
 os to mean OpenStack. This is confusing and also incorrect[0].
 
 ...

I have seen OS for OpenStack from the start.  Just look at the 
environment variables that the CLI reads.

Regards,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] OS or os are not acronyms for OpenStack

2014-08-15 Thread Mike Spreitzer
Anita Kuno ante...@anteaya.info wrote on 08/15/2014 01:08:44 PM:

 ...
 I think you hit the nail on the head here, Russell, it's fine in the
 right context.
 
 The definition of the right context however is somewhat elusive. I have
 chosen (it is my own fault) to place myself in the area where the folks
 I deal with struggle with understanding context. The newcomers to the
 third party space and folks creating stackforge repos don't have the
 benefit of the understanding that core reviewers have (would I be
 accurate in saying that it is mostly nova reviewers who have responded
 to my initial post thus far?).

I suffered from an instance of this confusion myself when I was just 
getting started,
and have seen colleagues get confused too.  I suspect this problem hits 
many
newcomers.

Regards,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] OS or os are not acronyms for OpenStack

2014-08-15 Thread Mike Spreitzer
Russell Bryant rbry...@redhat.com wrote on 08/15/2014 01:49:40 PM:
 ... 
 but surely when it comes to learning OpenStack itself, the OpenStack
 community, dev processes, tools, etc  this has got to be extremely
 far down the list of barriers to entry.

No argument there.  I am spending decimal orders of magnitude more time 
looking for a working concrete example of using DevStack to install 
OpenStack with Neutron inside a VM so that the nested VMs can talk to the 
outside world.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Neutron][DevStack] How to increase developer usage of Neutron

2014-08-14 Thread Mike Spreitzer
I'll bet I am not the only developer who is not highly competent with 
bridges and tunnels, Open VSwitch, Neutron configuration, and how DevStack 
transmutes all those.  My bet is that you would have more developers using 
Neutron if there were an easy-to-find and easy-to-follow recipe to use, to 
create a developer install of OpenStack with Neutron.  One that's a pretty 
basic and easy case.  Let's say a developer gets a recent image of Ubuntu 
14.04 from Canonical, and creates an instance in some undercloud, and that 
instance has just one NIC, at 10.9.8.7/16.  If there were a recipe for 
such a developer to follow from that point on, it would be great.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron][DevStack] How to increase developer usage of Neutron

2014-08-14 Thread Mike Spreitzer
CARVER, PAUL pc2...@att.com wrote on 08/14/2014 09:35:17 AM:

 Mike Spreitzer [mailto:mspre...@us.ibm.com] wrote:
 
 I'll bet I am not the only developer who is not highly competent with
 bridges and tunnels, Open VSwitch, Neutron configuration, and how 
DevStack
 transmutes all those.  My bet is that you would have more developers 
using
 Neutron if there were an easy-to-find and easy-to-follow recipe to use, 
to
 create a developer install of OpenStack with Neutron.  One that's a 
pretty
 basic and easy case.  Let's say a developer gets a recent image of 
Ubuntu
 14.04 from Canonical, and creates an instance in some undercloud, and 
that
 instance has just one NIC, at 10.9.8.7/16.  If there were a recipe for
 such a developer to follow from that point on, it would be great.
 
 https://wiki.openstack.org/wiki/NeutronDevstack worked for me.
 
 However, I'm pretty sure it's only a single node all in one setup. At 
least,
 I created only one VM to run it on and I don't think DevStack has 
created
 multiple nested VMs inside of the one I create to run DevStack. I 
haven't
 gotten around to figuring out how to setup a full multi-node DevStack
 setup with separate compute nodes and network nodes and GRE/VXLAN 
tunnels.
 
 There are multi-node instructions on that wiki page but I haven't tried
 following them. If someone has a Vagrant file that creates a full multi-
 node Neutron devstack complete with GRE/VXLAN tunnels it would be great
 if they could add it to that wiki page.

A working concrete recipe for a single-node install would be great.

https://wiki.openstack.org/wiki/NeutronDevstack is far from a concrete 
recipe, leaving many blanks to be filled in by the reader.
My problem is that as a non-expert in the relevant networking arcana, 
Neutron implementation,
and DevStack configuration options, it is not entirely obvious how to fill 
in the blanks.
For starters, 
http://docs.openstack.org/admin-guide-cloud/content/network-connectivity.html
speaks of four networks and, appropriately for a general page like that, 
does not relate them to NICs.
But at the start of the day, I need to know how many NICs to put on my 
host VM (the one in which I
will run DevStack to install OpenStack), how to configure them in the host 
VM's operating system,
and how to tell DevStack whatever details it needs and cannot figure out 
on its own
(I am not even clear on what that set is).
I need to know how to derive the fixed and floating IP address ranges from 
the networking context
of my host VM.
A recipe that requires more than one NIC on my host VM can be problematic 
in some situations,
which is why I suggested starting with a recipe for a host with a single 
NIC.

I am not using Xen in my undercloud.  I suspect many developers are not 
using Xen.
I did not even know it was possible to install OpenStack inside a Xen VM;
does that still work?

I was hoping for a working concrete recipe that does not depend on the 
undercloud,
rather something that works in a vanilla context that can easily be 
established with
whatever undercloud a given developer is using.

Thanks,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova][Neutron][Technical Committee] nova-network - Neutron. Throwing a wrench in the Neutron gap analysis

2014-08-05 Thread Mike Spreitzer
Monty Taylor mord...@inaugust.com wrote on 08/05/2014 12:27:14 PM:

 On 08/05/2014 09:18 AM, Jay Pipes wrote:
  Hello stackers, TC, Neutron contributors,
 
  At the Nova mid-cycle meetup last week in Oregon, during the 
discussion
  about the future of nova-network, the topic of nova-network - Neutron
  migration came up.
 
  For some reason, I had been clueless about the details of one of the
  items in the gap analysis the TC had requested [1]. Namely, the 5th
  item, about nova-network - Neutron migration, which is detailed in 
the
  following specification:
 
  
https://review.openstack.org/#/c/101921/12/specs/juno/neutron-migration.rst

 
  ...
 
  I personally believe that this requirement to support a live migration
  with no downtime of running instances between a nova-network and a
  Neutron deployment *is neither realistic, nor worth the extensive time
  and technical debt needed to make this happen*.
 
  I suggest that it would be better to instead provide good instructions
  for doing cold migration (snapshot VMs in old nova-network deployment,
  store in Swift or something, then launch VM from a snapshot in new
  Neutron deployment) -- which should cover the majority of deployments 
--
  and then write some instructions for what to look out for when doing a
  custom migration for environments that simply cannot afford any 
downtime
  and *really* want to migrate to Neutron. For these deployments, it's
  almost guaranteed that they will need to mangle their existing 
databases
  and do manual data migration anyway -- like RAX did when moving from
  nova-network to Neutron. The variables are too many to list here, and
  the number of deployments actually *needing* this work seems to me to 
be
  very limited. Someone suggested Metacloud *might* be the only 
deployment
  that might meet the needs for a live nova-network - Neutron 
migration.
  Metacloud folks, please do respond here!
 
  ...
 
 I agree 100%. Although I understand the I think it's an unreasonably 
 high burden in an area where there are many many other real pressing 
 issues that need to be solved.

I will go a little further.  My focus is on workloads that are composed of 
scaling groups (one strict way of saying cattle not pets).  In this case 
I do not need to migrate individual Compute instances, just shut down 
obsolete ones and start shiny new ones.

Regards,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Networking Docs Swarm - Brisbane 9 August

2014-08-04 Thread Mike Spreitzer
Lana Brindley openst...@lanabrindley.com wrote on 08/04/2014 11:05:24 
PM:
 I just wanted to let you all know about the OpenStack Networking Docs 
 Swarm being held in Brisbane on 9 August.
 ...

+++ on this.

I can not contribute answers, but have lots of questions.

Let me suggest that documentation is needed both for cloud providers doing 
general deployment and also for developers using DevStack.  Not all of us 
developers are Neutron experts, so we need decent documentation.  And 
developers sometimes need to use host machines with fewer than the ideal 
number of NICs.  Sometimes those host machines are virtual, leading to 
nested virtualization (of network as well as compute).

Thanks!
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [neutron] API confusion

2014-08-01 Thread Mike Spreitzer
http://developer.openstack.org/api-ref-networking-v2.html and 
http://docs.openstack.org/api/openstack-network/2.0/content/GET_listMembers__v2.0_pools__pool_id__members_lbaas_ext_ops_member.html
 
say that to list LB pool members, the URL to GET is 
/v2.0/pools/{pool_id}/members

When I use the CLI (`neutron -v lb-member-list`) I see a GET on 
/v2.0/lb/members.json

What's going on here?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [ceilometer][nova] extra Ceilometer Samples of the instance gauge

2014-07-30 Thread Mike Spreitzer
In a normal DevStack install, each Compute instance causes one Ceilometer 
Sample every 10 minutes.  Except, there is an extra one every hour.  And a 
lot of extra ones at the start.  What's going on here?

For example:

$ ceilometer sample-list -m instance -q 
resource=9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab
+--+--+---++--++
| Resource ID  | Name | Type  | Volume | Unit  
  | Timestamp  |
+--+--+---++--++
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T18:09:28|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T18:00:54.009877 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T17:59:28|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T17:49:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T17:39:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T17:29:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T17:19:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T17:09:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T17:00:07.002075 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T16:59:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T16:49:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T16:39:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T16:29:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T16:19:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T16:09:26|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T16:00:20.172520 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T15:59:27|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T15:49:26|
...
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T05:19:21|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T05:09:21|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T05:00:41.909634 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:59:21|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:49:21|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:39:21|
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:55.049799 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:54.834377 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:51.905095 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:40.962977 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:40.201907 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:40.091348 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:39.858939 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:39.693631 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:39.523561 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:39.421295 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:39.411968 |
| 9108b64e-0e30-45fa-9fdf-ccc2b9cf8dab | instance | gauge | 1.0| 
instance | 2014-07-30T04:32:38.916604 |
+--+--+---++--++

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] objects notifications

2014-07-29 Thread Mike Spreitzer
Gary Kotton gkot...@vmware.com wrote on 07/29/2014 12:43:08 PM:

 Hi,
 When reviewing https://review.openstack.org/#/c/107954/ it occurred 
 to me that maybe we should consider having some kind of generic 
 object wrapper that could do notifications for objects. Any thoughts on 
this?

I am not sure what that would look like, but I agree that we have a 
problem with too many things not offering notifications.  If there were 
some generic way to solve that problem, it would indeed be great.

Thanks,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-24 Thread Mike Spreitzer
Doug Wiegley do...@a10networks.com wrote on 07/23/2014 11:24:32 PM:

 Hi Mike,
 
  and listed the possible values of the status field, including 
 INACTIVE.  Other sources are telling me that status=INACTIVE when 
 the health monitor thinks the member is unhealthy, status!=INACTIVE 
 when the health monitor thinks the member is healthy.  What's going on 
here? 
 
 Indeed, the code will return a server status of INACTIVE if the 
 lbaas agent marks a member ‘DOWN’.  But, nowhere can I find that it 
 actually ever does so.
 
 My statements about the status field for lbaas/neutron came from the
 author of the ref lbaas driver; I’ll check with him tomorrow and see
 if I misunderstood.

I did an experiment, and found that the PoolMember status did indeed 
switch between ACTIVE and INACTIVE depending on the health of the member.

Thanks,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [heat] Convergence question - notifications?

2014-07-24 Thread Mike Spreitzer
The Convergence spec speaks of adding notification (
http://docs-draft.openstack.org/94/96394/3/gate/gate-heat-specs-docs/e3e9e06/doc/build/html/specs/convergence.html#proposed-change
).  What sort of notifications are these?  Are they something to which the 
implementation of a scaling group could subscribe, to learn when a member 
has gone missing?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-23 Thread Mike Spreitzer
Doug Wiegley do...@a10networks.com wrote on 07/16/2014 04:58:52 PM:

 You do recall correctly, and there are currently no mechanisms for
 notifying anything outside of the load balancer backend when the health
 monitor/member state changes.

But there *is* a mechanism for some outside thing to query the load 
balancer for the health of a pool member, right?  I am thinking 
specifically of 
http://docs.openstack.org/api/openstack-network/2.0/content/GET_showMember__v2.0_pools__pool_id__members__member_id__lbaas_ext_ops_member.html
 
--- whose response includes a status field for the member.  Is there 
documentation for what values can appear in that field, and what each 
value means?

Supposing we can leverage the pool member status, there remains an issue: 
establishing a link between an OS::Neutron::PoolMember and the 
corresponding scaling group member.  We could conceivably expand the 
scaling group code so that if the member type is a stack then the contents 
of the stack are searched (perhaps recursively) for resources of type 
OS::Neutron::PoolMember, but that is a tad too automatic for my taste.  It 
could pick up irrelevant PoolMembers.  And such a level of implicit 
behavior is outside our normal style of doing things.

We could follow the AWS style, by adding an optional property to the 
scaling group resource types --- where the value of that property can be 
the UUID of an OS::Neutron::LoadBalancer or an OS::Neutron::Pool.  But 
that still does not link up an individual scaling group member with its 
corresponding PoolMember.

Remember that if we are doing this at all, each scaling group member must 
be a stack.  I think the simplest way to solve this would be to define a 
way that a such stack can put in its outputs the ID of the corresponding 
PoolMember.  I would be willing to settle for simply saying that if such a 
stack has an output of type string and name __OS_pool_member then the 
value of that output is taken to be the ID of the corresponding 
PoolMember.  Some people do not like reserved names; if that must be 
avoided then we can expand the schema language with a way to identify 
which stack output carries the PoolMember ID.  Another alternative would 
be to add an optional scaling group property to carry the name of the 
stack output in question.

 There is also currently no way for an external system to inject health
 information about an LB or its members.

I do not know that the injection has to be to the LB; in AWS the injection 
is to the scaling group.  That would be acceptable to me too.

Thoughts?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-23 Thread Mike Spreitzer
Doug Wiegley do...@a10networks.com wrote on 07/23/2014 03:43:02 PM:

 From: Doug Wiegley do...@a10networks.com
 ...
 The state of the world today: ‘status’ in the neutron database is 
 configuration/provisioning status, not operational status.  Neutron-
 wide thing.  We were discussing adding operational status fields (or
 a neutron REST call to get the info from the backend) last month, 
 but it’s something that isn’t planned for a serious conversation 
 until Kilo, at present.

Thanks for the prompt response.  Let me just grasp at one last straw: is 
there any chance that Neutron will soon define and implement Ceilometer 
metrics that reveal PoolMember health?

Thanks,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-23 Thread Mike Spreitzer
Stephen Balukoff sbaluk...@bluebox.net wrote on 07/23/2014 09:14:35 PM:

 It's probably worth pointing out that most of the Neutron LBaaS team
 are spending most of our time doing a major revision to Neutron 
 LBaaS. How stats processing should happen has definitely been 
 discussed but not resolved at present-- and in any case it was 
 apparent to those working on the project that it has secondary 
 importance compared to the revision work presently underway.
 
 I personally would like to have queries about most objects in the 
 stats API to Neutron LBaaS return a dictionary or

I presume you meant of rather than or.

   statuses for child
 objects which then a UI or auto-scaling system can interpret however
 it wishes.

That last part makes me a little nervious.  I have seen can interpret 
however it wishes mean can not draw any useful inferences because there 
are no standards for that content.

I presume that as the grand and glorious future arrives, it will be with 
due respect for backwards compatibility.

In the present, I am getting what appears to be conflicting information on 
the status field of the responses of 
http://docs.openstack.org/api/openstack-network/2.0/content/GET_showMember__v2.0_pools__pool_id__members__member_id__lbaas_ext_ops_member.html

Doug Wiegely wrote
 ‘status’ in the neutron database is configuration/provisioning status, 
not operational status
and listed the possible values of the status field, including INACTIVE. 
Other sources are telling me that status=INACTIVE when the health monitor 
thinks the member is unhealthy, status!=INACTIVE when the health monitor 
thinks the member is healthy.  What's going on here?

Thanks,
Mike


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-18 Thread Mike Spreitzer
Thomas Herve thomas.he...@enovance.com wrote on 07/17/2014 02:06:13 AM:

 There are 4 resources related to neutron load balancing. 
 OS::Neutron::LoadBalancer is probably the least useful and the one 
 you can *not* use, as it's only there for compatibility with 
 AWS::AutoScaling::AutoScalingGroup. OS::Neutron::HealthMonitor does 
 the health checking part, although maybe not in the way you want it.

OK, let's work with these.  My current view is this: supposing the 
Convergence work delivers monitoring of health according to a member's 
status in its service and reacts accordingly, the gaps (compared to AWS 
functionality) are the abilities to (1) get member health from 
application level pings (e.g., URL polling) and (2) accept member health 
declarations from an external system, with consistent reaction to health 
information from all sources.

Source (1) is what an OS::Neutron::HealthMonitor specifies, and an 
OS::Neutron::Pool is the thing that takes such a spec.  So we could 
complete the (1) part if there were a way to tell a scaling group to poll 
the member health information developed by an OS::Neutron::Pool.  Does 
that look like the right approach?

For (2), this would amount to having an API that an external system (with 
proper authorization) can use to declare member health.  In the grand and 
glorious future when scaling groups have true APIs rather than being Heat 
hacks, such a thing would be part of those APIs.  In the immediate future 
we could simply add this to the Heat API.  Such an operation would take 
somethings like a stack name or UUID, the name or UUID of a resource that 
is a scaling group, and the member name or UUID of the Resource whose 
health is being declared, and health_status=unhealthy.  Does that look 
about right?

For both of these new sources, the remaining question is how to get the 
right reaction.  In the case that the member is actually deleted already, 
life is easy.  Let's talk about the other cases.  Note that AWS admits 
that there might be false detection of unhealth as a member's contents 
finish getting into regular operation; AWS handles this by saying that the 
right reaction is to react only after unhealth has been consistently 
detected for a configured amount of time.  The simplest thing for a 
scaling group to do might be to include that hysteresis and eventually 
effect removal of a member by generating a new template that excludes the 
to-be-deleted member and doing an UPDATE on itself (qua stack) with that 
new template.  Does that look about right?

Thanks,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-18 Thread Mike Spreitzer
Clint Byrum cl...@fewbar.com wrote on 07/18/2014 12:56:32 PM:

 Excerpts from Mike Spreitzer's message of 2014-07-18 09:12:21 -0700:
  ...
  OK, let's work with these.  My current view is this: supposing the 
  Convergence work delivers monitoring of health according to a member's 

  status in its service and reacts accordingly, the gaps (compared to 
AWS 
  functionality) are the abilities to (1) get member health from 
  application level pings (e.g., URL polling) and (2) accept member 
health 
  declarations from an external system, with consistent reaction to 
health 
  information from all sources.
  
 
 Convergence will not deliver monitoring, though I understand how one
 might have that misunderstanding. Convergence will check with the API
 that controls a physical resource to determine what Heat should consider
 its status to be for the purpose of ongoing orchestration.

If I understand correctly, your point is that healing is not automatic. 
Since a scaling group is a nested stack, the observing part of Convergence 
will automatically note in the DB when the physical resource behind a 
scaling group member (in its role as a stack resource) is deleted.  And 
when convergence engine gets around to acting on that Resource, the 
backing physical resource will be automatically re-created.  But there is 
nothing that automatically links the notice of divergence to the 
converging action.  Have I got that right?

  Source (1) is what an OS::Neutron::HealthMonitor specifies, and an 
  OS::Neutron::Pool is the thing that takes such a spec.  So we could 
  complete the (1) part if there were a way to tell a scaling group to 
poll 
  the member health information developed by an OS::Neutron::Pool.  Does 

  that look like the right approach?
  
  For (2), this would amount to having an API that an external system 
(with 
  proper authorization) can use to declare member health.  In the grand 
and 
  glorious future when scaling groups have true APIs rather than being 
Heat 
  hacks, such a thing would be part of those APIs.  In the immediate 
future 
  we could simply add this to the Heat API.  Such an operation would 
take 
  somethings like a stack name or UUID, the name or UUID of a resource 
that 
  is a scaling group, and the member name or UUID of the Resource whose 
  health is being declared, and health_status=unhealthy.  Does that 
look 
  about right?
  
 
 Isn't (2) covered already by the cloudwatch API in Heat? I am going to
 claim ignorance of it a bit, as I've never used it, but it seems like
 the same thing.

I presume that by cloudwatch API you mean Ceilometer.  Today a 
Ceilometer alarm can be given a URL to invoke but can not be told about 
any special headers or body to use in the invocation (i.e., no parameters 
for the HTTP operation).

More to the point, the idea here is supporting a general external system 
that might determine health in its own way, not necessarily through 
programming Ceilometer to detect it.

Thanks,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-16 Thread Mike Spreitzer
Clint Byrum cl...@fewbar.com wrote on 07/02/2014 01:54:49 PM:

 Excerpts from Qiming Teng's message of 2014-07-02 00:02:14 -0700:
  Just some random thoughts below ...
  
  On Tue, Jul 01, 2014 at 03:47:03PM -0400, Mike Spreitzer wrote:
   In AWS, an autoscaling group includes health maintenance 
 functionality --- 
   both an ability to detect basic forms of failures and an abilityto 
react 
   properly to failures detected by itself or by a load balancer.  What 
is 
   the thinking about how to get this functionality in OpenStack? Since 

  
  We are prototyping a solution to this problem at IBM Research - China
  lab.  The idea is to leverage oslo.messaging and ceilometer events for
  instance (possibly other resource such as port, securitygroup ...)
  failure detection and handling.
  
 
 Hm.. perhaps you should be contributing some reviews here as you may
 have some real insight:
 
 https://review.openstack.org/#/c/100012/
 
 This sounds a lot like what we're working on for continuous convergence.

I noticed that health checking in AWS goes beyond convergence.  In AWS an 
ELB can be configured with a URL to ping, for application-level health 
checking.  And an ASG can simply be *told* the health of a member by a 
user's own external health system.  I think we should have analogous 
functionality in OpenStack.  Does that make sense to you?  If so, do you 
have any opinion on the right way to integrate, so that we do not have 
three completely independent health maintenance systems?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-16 Thread Mike Spreitzer
Doug Wiegley do...@a10networks.com wrote on 07/16/2014 04:58:52 PM:

 On 7/16/14, 2:43 PM, Clint Byrum cl...@fewbar.com wrote:
 
 Excerpts from Mike Spreitzer's message of 2014-07-16 10:50:42 -0700:
 ...
  I noticed that health checking in AWS goes beyond convergence.  In 
AWS
 an 
  ELB can be configured with a URL to ping, for application-level 
health
  checking.  And an ASG can simply be *told* the health of a member by 
a
  user's own external health system.  I think we should have analogous
  functionality in OpenStack.  Does that make sense to you?  If so, do
 you 
  have any opinion on the right way to integrate, so that we do not 
have
  three completely independent health maintenance systems?
 
 The check url is already a part of Neutron LBaaS IIRC.

Yep.  LBaaS is a work in progress, right?  Those of use using Nova 
networking are not feeling the love, unfortunately.

As far as Heat goes, there is no LBaaS resource type.  The 
OS::Neutron::LoadBalancer resource type does not have any health checking 
properties.  The AWS::ElasticLoadBalancing::LoadBalancer does have a 
parameter that prescribes health checking --- but, as far as I know, there 
is no way to ask such a load balancer for its opinion of a member's 
health.

 What may not be
 a part is notifications for when all members are reporting down (which
 might be something to trigger scale-up).

I do not think we want an ASG to react only when all members are down;
I think an ASG should maintain at least its minimum size
(although I have to admit that I do not understand why the current code
has an explicit exception to that).

 You do recall correctly, and there are currently no mechanisms for
 notifying anything outside of the load balancer backend when the health
 monitor/member state changes.

This is true in AWS as well.  The AWS design is that you can configure the 
ASG to poll the ELB for its opinion of member health.  The idea seems to 
be that an ASG can get health information from three kinds of sources 
(polling status in EC2, polling ELB, and being explicitly informed), 
synthesizes its own summary opinion, and reacts in due time.

 There is also currently no way for an external system to inject health
 information about an LB or its members.
 
 Both would be interesting additions.
 
 doug
 
 
 
 If we don't have push checks in our auto scaling implementation then we
 don't have a proper auto scaling implementation.

I am not sure what is meant by push checks.

Thanks,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] autoscaling across regions and availability zones

2014-07-11 Thread Mike Spreitzer
Zane Bitter zbit...@redhat.com wrote on 07/10/2014 05:57:14 PM:

 On 09/07/14 22:38, Mike Spreitzer wrote:
  Zane Bitter zbit...@redhat.com wrote on 07/01/2014 06:54:58 PM:
 
On 01/07/14 16:23, Mike Spreitzer wrote:
...
 
 Hmm, now that I think about it, CloudFormation provides a Fn::GetAZs 
 function that returns a list of available AZs. That suggests an 
 implementation where you can specify an AZ

If we're whittling down then it would be one or more AZs, right?

when creating the stack and 
 the function returns only that value within that stack (and its 
 children). There's no way in OS::Heat::AutoScalingGroup to specify an 
 intrinsic function that is resolved in the context of the scaling 
 group's nested stack,

I am not sure I understand what you mean.  Is it: there is no way for the 
implementation of a resource type to create or modify an intrinsic 
function?

   but if the default value of the AZ for 
 OS::Nova::Server were calculated the same way then the user would have 
 the option of omitting the AZ (to allow the autoscaling implementation 
 to control it)

I am not sure I get this part.  If the scaling group member type is a 
Compute instance (as well as if it is not) then the template generated by 
the group (to implement the group) wants to put different resources in 
different AZs.  The nested stack that is the scaling group is given a 
whole list of AZs as its list-of-AZs parameter value.

or overriding it explicitly. At that point you don't even 

 need the intrinsic function.
 
 So don't assign a stack to a particular AZ as such, but allow the list 
 of valid AZs to be whittled down as you move toward the leaves of the 
 tree of templates.

I partially get the suggestion.  Let me repeat it back to see if it sounds 
right.
Let the stack create and update operations gain an optional parameter that 
is a list of AZs,
(noting that a stack operation parameter is something different from a 
parameter
specified in a template)
constrained to be a subset of the AZs available to the user in Heat's 
configured region;
the default value is the list of all AZs available to the user in Heat's 
configured region.
Redefine the Fn::GetAZs intrinsic to return that new parameter's value.
For each resource type that can be given a list of AZs, we (as plugin 
authors) redefine the default to be the list returned by Fn::GetAZs;
for each resource type that can be given a single AZ, we (as plugin 
authors) redefine the default to be one (which one?) of the AZs returned 
by Fn::GetAZs.
That would probably require some finessing around the schema technology, 
because a parameter's default value is fixed when the resource type is 
registered, right?
A template generated by scaling group code somehow uses that new stack 
operation parameter
to set the member's AZ when the member is a stack and the scaling group is 
spanning AZs.
Would the new stack operation parameter (list of AZs) be reflected as a 
property of OS::Heat::Stack?
How would that list be passed in a scenario like 
https://review.openstack.org/#/c/97366/10/hot/asg_of_stacks.yaml,unified
where the member type is a template filename and the member's properties 
are simply the stack's parameters?
Can the redefinitions mentioned here be a backward compatibility problem?

So yes, the tricky part is how to handle that when the scaling unit 
is
not a server (or a provider template with the same interface as a
  server).
   
One solution would have been to require that the scaled unit was,
indeed, either an OS::Nova::Server or a provider template with the 
same
interface as (or a superset of) an OS::Nova::Server, but the 
consensus
was against that. (Another odd consequence of this decision is that
we'll potentially be overwriting an AZ specified in the launch 
config
section with one from the list supplied to the scaling group 
itself.)
   
For provider templates, we could insert a pseudo-parameter 
containing
the availability zone. I think that could be marginally better than
taking over one of the user's parameters, but you're basically on 
the
right track IMO.
 
  I considered a built-in function or pseudo parameter and rejected them
  based on a design principle that was articulated in an earlier
  discussion: no modes.  Making the innermost template explicitly
  declare that it takes an AZ parameter makes it more explicit what is
  going on.  But I agree that this is a relatively minor design point, 
and
  would be content to go with a pseudo-parameter if the community really
  prefers that.
 
Unfortunately, that is not the end of the story, because we still 
have
to deal with other types of resources being scaled. I always 
advocated
for an autoscaling resource where the scaled unit was either a 
provider
stack (if you provided a template) or an OS::Nova::Server (if you
didn't

Re: [openstack-dev] 答复: [heat] autoscaling across regions and availability zones

2014-07-09 Thread Mike Spreitzer
Huangtianhua huangtian...@huawei.com wrote on 07/04/2014 02:35:56 AM:

 I have register a bp about this : https://blueprints.launchpad.net/
 heat/+spec/implement-autoscalinggroup-availabilityzones
 ・ 
 ・ And I am thinking how to implement this recently.
 ・ 
 ・ According to AWS autoscaling implementation  “attempts to 
 distribute instances evenly between the Availability Zones that are 
 enabled for your Auto Scaling group. 
 ・ Auto Scaling does this by attempting to launch new 
 instances in the Availability Zone with the fewest instances. If the
 attempt fails, however, Auto Scaling will attempt to launch in other
 zones until it succeeds.”
 
 But there is a doubt about the “fewest instance”, .e.g 
 
 There are two azs,
Az1: has two instances
Az2: has three instances
 ・And then to create a asg with 4 instances, I think we 
 should create two instances respectively in az1 and az2, right? Now 
 if need to extend to 5 instances for the asg, which az to lauch new 
instance?
 If you interested in this bp, I think we can discuss thisJ

The way AWS handles this is described in 
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AS_Concepts.html#arch-AutoScalingMultiAZ

That document leaves a lot of freedom to the cloud provider.  And 
rightfully so, IMO.  To answer your specific example, when spreading 5 
instances across 2 zones, the cloud provider gets to pick which zone gets 
3 and which zone gets 2.  As for what a Heat scaling group should do, that 
depends on what Nova can do for Heat.  I have been told that Nova's 
instance-creation operation takes an optional parameter that identifies 
one AZ and, if that parameter is not provided, then a configured default 
AZ is used.  Effectively, the client has to make the choice.  I would 
start out with Heat making a random choice; in subsequent development it 
might query or monitor Nova for some statistics to guide the choice.

An even more interesting issue is the question of choosing which member(s) 
to remove when scaling down.  The approach taken by AWS is documented at 
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/us-termination-policy.html
but the design there has redundant complexity and the doc is not well 
written.  Following is a short sharp presentation of an isomorphic system.
A client that owns an ASG configures that ASG to have a series (possibly 
empty) of instance termination policies; the client can change the series 
during the ASG's lifetime.  Each policy is drawn from the following menu:
OldestLaunchConfiguration
ClosestToNextInstanceHour
OldestInstance
NewestInstance
(see the AWS doc for the exact meaning of each).  The signature of a 
policy is this: given a set of candidate instances for removal, return a 
subset (possibly the whole input set).
When it is time to remove instances from an ASG, they are chosen one by 
one.  AWS uses the following procedure to choose one instance to remove. 
1.  Choose the AZ from which the instance will be removed.  The choice 
is based primarily on balancing the number of group members in each AZ, 
and ties are broken randomly.
2.  Starting with a candidate set consisting of all the ASG's 
members in the chosen AZ, run the configured series of policies to 
progressively narrow down the set of candidates.
3.  Use OldestLaunchConfiguration and then ClosestToNextInstanceHour 
to further narrow the set of candidates.
4.  Make a random choice among the final set of candidates.
Since each policy returns its input when its input's size is 1 we do not 
need to talk about early exits when defining the procedure (although the 
implementation might make such optimizations).
I plan to draft a spec.
Thanks,
Mike


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] autoscaling across regions and availability zones

2014-07-09 Thread Mike Spreitzer
Zane Bitter zbit...@redhat.com wrote on 07/01/2014 06:54:58 PM:

 On 01/07/14 16:23, Mike Spreitzer wrote:
  An AWS autoscaling group can span multiple availability zones in one
  region.  What is the thinking about how to get analogous functionality
  in OpenStack?
 ...
  Currently, a stack does not have an AZ.  That makes the case of an
  OS::Heat::AutoScalingGroup whose members are nested stacks interesting
  --- how does one of those nested stacks get into the right AZ?  And 
what
  does that mean, anyway?  The meaning would have to be left up to the
  template author.  But he needs something he can write in his member
  template to reference the desired AZ for the member stack.  I suppose 
we
  could stipulate that if the member template has a parameter named
  availability_zone and typed string then the scaling group takes 
care
  of providing the right value to that parameter.
 
 The concept of an availability zone for a stack is not meaningful. 
 Servers have availability zones; stacks exist in one region. It is up to 

 the *operator*, not the user, to deploy Heat in such a way that it 
 remains highly-available assuming the Region is still up.

There are two distinct issues there: (1) making the heat engine HA and (2) 
making a scaling group of stacks span across AZs (within a region).  I 
agree that (1) is the cloud provider's problem, and never meant to suggest 
otherwise.  I think (2) makes sense by analogy: a nested stack is a way of 
implementing a particular abstraction (defined by the template) --- in 
fact the outer template author might not even be aware that the group 
members are stacks, thanks to provider templates --- and here we suppose 
the user has chosen to use an abstraction that makes sense to be 
considered to be in an AZ.  While a stack in general does not have an 
AZ, I think we can suppose that if the outer template author asked for 
stacks to be spread across AZs then the stacks in question can reasonably 
considered to each be in one AZ.  For example, the inner template might 
contain a Compute instance and a Cinder volume and an attachment between 
the two; such a stack makes sense to put in an AZ.  Heat itself does not 
even need there to be any particular real meaning to a stack being in an 
AZ, all I am proposing is that Heat make this concept available to the 
authors of the outer and innermost templates to use in whatever way they 
find useful.

 So yes, the tricky part is how to handle that when the scaling unit is 
 not a server (or a provider template with the same interface as a 
server).
 
 One solution would have been to require that the scaled unit was, 
 indeed, either an OS::Nova::Server or a provider template with the same 
 interface as (or a superset of) an OS::Nova::Server, but the consensus 
 was against that. (Another odd consequence of this decision is that 
 we'll potentially be overwriting an AZ specified in the launch config 
 section with one from the list supplied to the scaling group itself.)
 
 For provider templates, we could insert a pseudo-parameter containing 
 the availability zone. I think that could be marginally better than 
 taking over one of the user's parameters, but you're basically on the 
 right track IMO.

I considered a built-in function or pseudo parameter and rejected them 
based on a design principle that was articulated in an earlier discussion: 
no modes.  Making the innermost template explicitly declare that it 
takes an AZ parameter makes it more explicit what is going on.  But I 
agree that this is a relatively minor design point, and would be content 
to go with a pseudo-parameter if the community really prefers that.

 Unfortunately, that is not the end of the story, because we still have 
 to deal with other types of resources being scaled. I always advocated 
 for an autoscaling resource where the scaled unit was either a provider 
 stack (if you provided a template) or an OS::Nova::Server (if you 
 didn't), but the implementation that landed followed the design of 
 ResourceGroup by allowing (actually, requiring) you to specify an 
 arbitrary resource type.
 
 We could do something fancy here involving tagging the properties schema 

 with metadata so that we could allow plugin authors to map the AZ list 
 to an arbitrary property. However, I propose that we just raise a 
 validation error if the AZ is specified for a resource that is not 
 either an OS::Nova::Server or a provider template.

Yes, I agree with limiting ambition.  For OS::Heat::AutoScalingGroup and 
ResourceGroup, I think a pretty simple rule will cover all the cases that 
matter besides nested stack: if the member resource type has a string 
parameter named availability zone (camel or snake case) then this is 
valid.

I just reviewed LaunchConfiguration (
http://docs.openstack.org/developer/heat/template_guide/cfn.html#AWS::AutoScaling::LaunchConfiguration
) and noticed that it does not have an availability_zone.

Since AWS::AutoScalingGroup

Re: [openstack-dev] 答复: 答复: [heat] autoscaling across regions and availability zones

2014-07-09 Thread Mike Spreitzer
Huangtianhua huangtian...@huawei.com wrote on 07/09/2014 10:20:42 PM:

 发件人: Mike Spreitzer [mailto:mspre...@us.ibm.com] 
 发送时间: 2014年7月10日 3:19
 收件人: OpenStack Development Mailing List (not for usage questions)
 主题: Re: [openstack-dev] 答复: [heat] autoscaling across regions 
 and availability zones
 
 Huangtianhua huangtian...@huawei.com wrote on 07/04/2014 02:35:56 AM:
 
  I have register a bp about this : https://blueprints.launchpad.net/
  heat/+spec/implement-autoscalinggroup-availabilityzones 
  ・ 
  ・ And I am thinking how to implement this recently. 
  ・ 
  ・ According to AWS autoscaling implementation  “attempts to 
  distribute instances evenly between the Availability Zones that are 
  enabled for your Auto Scaling group. 
  ・ Auto Scaling does this by attempting to launch new 
  instances in the Availability Zone with the fewest instances. If the
  attempt fails, however, Auto Scaling will attempt to launch in other
  zones until it succeeds.” 
  
  But there is a doubt about the “fewest instance”, .e.g 
  
  There are two azs, 
 Az1: has two instances 
 Az2: has three instances 
  ・And then to create a asg with 4 instances, I think we 
  should create two instances respectively in az1 and az2, right? Now 
  if need to extend to 5 instances for the asg, which az to lauch 
 new instance? 
  If you interested in this bp, I think we can discuss thisJ 
 
 The way AWS handles this is described in http://docs.aws.amazon.com/
 
AutoScaling/latest/DeveloperGuide/AS_Concepts.html#arch-AutoScalingMultiAZ 

 
 That document leaves a lot of freedom to the cloud provider.  And 
 rightfully so, IMO.  To answer your specific example, when spreading
 5 instances across 2 zones, the cloud provider gets to pick which 
 zone gets 3 and which zone gets 2.  As for what a Heat scaling group
 should do, that depends on what Nova can do for Heat.  I have been 
 told that Nova's instance-creation operation takes an optional 
 parameter that identifies one AZ and, if that parameter is not 
 provided, then a configured default AZ is used.  Effectively, the 
 client has to make the choice.  I would start out with Heat making a
 random choice; in subsequent development it might query or monitor 
 Nova for some statistics to guide the choice. 

 yes, I read the doc, as you said, the doc is not well 
 written, so I doubt about the “fewest instance” before, but now IMO,
 “fewest instance” means the instances of the group, so you are 
 right, to my specific example, the instance should be launch at 
 random or in a round-robin mode.
 
 An even more interesting issue is the question of choosing which 
 member(s) to remove when scaling down.  The approach taken by AWS is
 documented at http://docs.aws.amazon.com/AutoScaling/latest/
 DeveloperGuide/us-termination-policy.html 
 but the design there has redundant complexity and the doc is not 
 well written.  Following is a short sharp presentation of an 
 isomorphic system. 
 A client that owns an ASG configures that ASG to have a series 
 (possibly empty) of instance termination policies; the client can 
 change the series during the ASG's lifetime.  Each policy is drawn 
 from the following menu: 
 OldestLaunchConfiguration 
 ClosestToNextInstanceHour 
 OldestInstance 
 NewestInstance
 and there is a default policy of termination...

If you are objecting to the fact that I omitted Default from the menu of 
available policies, note that there is no need to discuss a Default 
policy.  Remember that doc also stipulates that if your list of policies 
includes Default then Default should be at the end of your list. Since 
the Default policy is always applied after your list (if there is a need), 
there is no need to explicitly declare it at the end.  Note also that the 
doc is confusing about default policy; at first it says that the default 
policy selects an AZ, and later it mentions using the default policy to 
narrow down a set of candidates within an AZ that has already been chosen. 
 So a simpler, equivalent explanation has no need to ever talk about a 
default policy, rather it can stipulate that the configured list of ways 
to narrow down the set of candidate instances is always followed by 
OldestLaunchConfiguration, then ClosestToNextInstanceHour, then a random 
choice (which is how the so-called default policy narrows down the set 
of candidates after the AZ has been chosen).

I brought this whole sub-topic up only to mostly dismiss it.  OpenStack 
scaling groups do not currently attempt anything similar to 
OldestLaunchConfiguration or ClosestToNextInstanceHour and I recommend 
that the work on spanning across AZs should not attempt to change that. My 
recommendation is that this spec only call for adding a balancing 
selection of AZ to what is currently done.  I have also opened a separate 
discussion (ML subject One more lifecycle plug point - in scaling groups
) about opening up this policy issue to give

Re: [openstack-dev] [heat] One more lifecycle plug point - in scaling groups

2014-07-03 Thread Mike Spreitzer
Steven Hardy sha...@redhat.com wrote on 07/02/2014 06:16:21 AM:

 On Wed, Jul 02, 2014 at 02:41:19AM +, Adrian Otto wrote:
 Zane,
 If you happen to have a link to this blueprint, could you replywith 
it? ...

 I believe Zane was referring to:
 
 https://blueprints.launchpad.net/heat/+spec/update-hooks
 
 This is also related to the action aware software config spec:
 
 https://review.openstack.org/#/c/98742/
 
 So in future, you might autoscale nested stacks containing action-aware
 software config resources, then you could define specific actions which
 happen e.g on scale-down (on action DELETE).

Thanks, those are great pointers.  The second pretty much covers the 
first, right?

I do think the issue these address --- the need to get application logic 
involved in, e.g., shutdown --- is most of what an application needs; 
involvement in selection of which member(s) to delete is much less 
important (provided that clean shutdown mechanism prevents concurrent 
shutdowns).  So that provides a pretty good decoupling between the 
application's concerns and a holistic scheduler's concerns.

Thanks,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-02 Thread Mike Spreitzer
Zane Bitter zbit...@redhat.com wrote on 07/01/2014 06:58:47 PM:

 On 01/07/14 15:47, Mike Spreitzer wrote:
  In AWS, an autoscaling group includes health maintenance functionality
  --- both an ability to detect basic forms of failures and an ability 
to
  react properly to failures detected by itself or by a load balancer.
What is the thinking about how to get this functionality in 
OpenStack?
Since OpenStack's OS::Heat::AutoScalingGroup has a more general 
member
  type, what is the thinking about what failure detection means (and how
  it would be accomplished, communicated)?
 
  I have not found design discussion of this; have I missed something?
 
 Yes :)
 
 https://review.openstack.org/#/c/95907/
 
 The idea is that Convergence will provide health maintenance for _all_ 
 forms of resources in Heat. Once this is implemented, autoscaling gets 
 it for free by virtue of that fact that it manages resources using Heat 
 stacks.

Ah, right.  My reading of that design is not quite so simple.  Note that 
in the User Stories section it calls for different treatment of Compute 
instances depending on whether they are in a scaling group.  That's why I 
was thinking of this from a scaling group perspective.  But perhaps the 
more natural approach is to take the pervasive perspective and figure out 
how to suppress convergence for the Compute instances to which it should 
not apply.

Thanks,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-02 Thread Mike Spreitzer
Mike Spreitzer/Watson/IBM@IBMUS wrote on 07/02/2014 02:41:48 AM:

 Zane Bitter zbit...@redhat.com wrote on 07/01/2014 06:58:47 PM:
 
  On 01/07/14 15:47, Mike Spreitzer wrote:
   In AWS, an autoscaling group includes health maintenance 
functionality
   --- both an ability to detect basic forms of failures and an ability 
to
   react properly to failures detected by itself or by a load balancer.
 What is the thinking about how to get this functionality in 
OpenStack?
 Since OpenStack's OS::Heat::AutoScalingGroup has a more general 
member
   type, what is the thinking about what failure detection means (and 
how
   it would be accomplished, communicated)?
  
   I have not found design discussion of this; have I missed something?
  
  Yes :)
  
  https://review.openstack.org/#/c/95907/
  
  The idea is that Convergence will provide health maintenance for _all_ 

  forms of resources in Heat. Once this is implemented, autoscaling gets 

  it for free by virtue of that fact that it manages resources using 
Heat 
  stacks.
 
 Ah, right.  My reading of that design is not quite so simple...

There are a couple more issues that arise in this approach.  The biggest 
one is how to integrate application-level failure detection.  I added a 
comment to this effect on the Convergence spec.

Another issue is that, at least initially, Convergence is not always on; 
rather, it is an operation that can be invoked on a stack.  When would a 
scaling group invoke this action on itself (more precisely, itself 
considered as a nested stack)?  One obvious possibility is on a periodic 
basis.  It the convergence operation is pretty cheap when no divergence 
has been detected, then that might be acceptable.  Otherwise we might want 
the scaling group to set up some sort of notification, but that would be 
another batch of member-type specific code.

Thanks,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-02 Thread Mike Spreitzer
Qiming Teng teng...@linux.vnet.ibm.com wrote on 07/02/2014 03:02:14 AM:

 Just some random thoughts below ...
 
 On Tue, Jul 01, 2014 at 03:47:03PM -0400, Mike Spreitzer wrote:
  ...
  I have not found design discussion of this; have I missed something?
  
  I suppose the natural answer for OpenStack would be centered around 
  webhooks... 
 
 Well, I would suggest we generalize this into a event messaging or
 signaling solution, instead of just 'webhooks'.  The reason is that
 webhooks as it is implemented today is not carrying a payload of useful
 information -- I'm referring to the alarms in Ceilometer.

OK, this is great (and Steve Hardy provided more details in his reply), I 
did not know about the existing abilities to have a payload.  However 
Ceilometer alarms are still deficient in that way, right?  A Ceilometer 
alarm's action list is simply a list of URLs, right?  I would be happy to 
say let's generalize Ceilometer alarms to allow a payload in an action.

 There are other cases as well.  A member failure could be caused by a 
 temporary communication problem, which means it may show up quickly when
 a replacement member is already being created.  It may mean that we have
 to respond to an 'online' event in addition to an 'offline' event?
 ...
 The problem here today is about the recovery of SG member.  If it is a
 compute instance, we can 'reboot', 'rebuild', 'evacuate', 'migrate' it,
 just to name a few options.  The most brutal way to do this is like what
 HARestarter is doing today -- delete followed by a create.

We could get into arbitrary subtlety, and maybe eventually will do better, 
but I think we can start with a simple solution that is widely applicable. 
 The simple solution is that once the decision has been made to do 
convergence on a member (note that this is distinct from merely detecting 
and noting a divergence) then it will be done regardless of whether the 
doomed member later appears to have recovered, and the convergence action 
for a scaling group member is to delete the old member and create a 
replacement (not in that order).

  When the member is a nested stack and Ceilometer exists, it could be 
the 
  member stack's responsibility to include a Ceilometer alarm that 
detects 
  the member stack's death and hit the member stack's deletion webhook. 
 
 This is difficult.  A '(nested) stack' is a Heat specific abstraction --
 recall that we have to annotate a nova server resource in its metadata
 to which stack this server belongs.  Besides the 'visible' resources
 specified in a template, Heat may create internal data structures and/or
 resources (e.g. users) for a stack.  I am not quite sure a stack's death
 can be easily detected from outside Heat.  It would be at least
 cumbersome to have Heat notify Ceilometer that a stack is dead, and then
 have Ceilometer send back a signal.

A (nested) stack is not only a heat-specific abstraction but its semantics 
and failure modes are specific to the stack (at least, its template).  I 
think we have no practical choice but to let the template author declare 
how failure is detected.  It could be as simple as creating a Ceilometer 
alarms that detect death one or more resources in the nested stack; it 
could be more complicated Ceilometer stuff; it could be based on something 
other than, or in addition to, Ceilometer.  If today there are not enough 
sensors to detect failures of all kinds of resources, I consider that a 
gap in telemetry (and think it is small enough that we can proceed 
usefully today, and should plan on filling that gap over time).

  There is a small matter of how the author of the template used to 
create 
  the member stack writes some template snippet that creates a 
Ceilometer 
  alarm that is specific to a member stack that does not exist yet. 
 
 How about just one signal responder per ScalingGroup?  A SG is supposed
 to be in a better position to make the judgement: do I have to recreate
 a failed member? am I recreating it right now or wait a few seconds?
 maybe I should recreate the member on some specific AZs?

That is confusing two issues.  The thing that is new here is making the 
scaling group recognize member failure; the primary reaction is to update 
its accounting of members (which, in the current code, must be done by 
making sure the failed member is deleted); recovery of other scaling group 
aspects is fairly old-hat, it is analogous to the problems that the 
scaling group already solves when asked to increase its size.

 ...
  I suppose we could stipulate that if the member template includes a 
  parameter with name member_name and type string then the OS OG 
takes 
  care of supplying the correct value of that parameter; as illustrated 
in 
  the asg_of_stacks.yaml of https://review.openstack.org/#/c/97366/ , a 
  member template can use a template parameter to tag Ceilometer data 
for 
  querying.  The URL of the member stack's deletion webhook could be 
passed 
  to the member

Re: [openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-02 Thread Mike Spreitzer
Steven Hardy sha...@redhat.com wrote on 07/02/2014 06:02:36 AM:

 On Wed, Jul 02, 2014 at 03:02:14PM +0800, Qiming Teng wrote:
  Just some random thoughts below ...
  
  On Tue, Jul 01, 2014 at 03:47:03PM -0400, Mike Spreitzer wrote:
   ...
 
 The resource signal interface used by ceilometer can carry whatever data
 you like, so the existing solution works fine, we don't need a new one 
IMO.

That's great, I did not know about it.  Thanks for the details on this, I 
do
think it is an improvement.  Yes, it is a slight security downgrade --- 
the
signed URL is effectively a capability, and allowing a payload increases
the cross section of what an attacker can do with one stolen capability.
But I am willing to live with it.

 ...
 Are you aware of the Existence of instance meter in ceilometer?

I am, and was thinking it might be very directly usable.  Has anybody 
tested or demonstrated this?

 ...
  How about just one signal responder per ScalingGroup?  A SG is 
supposed
  to be in a better position to make the judgement: do I have to 
recreate
  a failed member? am I recreating it right now or wait a few seconds?
  maybe I should recreate the member on some specific AZs?
 
 This is what we have already - you have one ScalingPolicy (which is a
 SignalResponder), and the ScalingPolicy is the place where you make the
 decision about what to do with the data provided from the alarm.

I think the existing ScalingPolicy is about a different issue. 
ScalingPolicy is mis-named; it is really only a ScalingAction, and it is 
about how to adjust the desired size.  It does not address the key missing 
piece here, which is how the scaling group updates its accounting of the 
number of members it has.  That accounting is done simply by counting 
members.  So if a member becomes dysfunctional but remains extant, the 
scaling group logic continues to count it.

Hmm, can a scaling group today properly cope with member deletion if 
prodded to do a ScalingPolicy(Action) that is 'add 1 member'?  (I had 
considered 'add 0 members' but that fails to produce a change in an 
important case --- when the size is now below minimum (fun fact about the 
code!). )

 ...
  I am not in favor of the per-member webhook design.  But I vote for an
  additional *implicit* parameter to a nested stack of any groups.  It
  could be an index or a name.
 
 I agree, we just need appropriate metadata in ceilometer, which can then 
be
 passed back to heat via the resource signal when the alarm happens.

We need to get the relevant meter samples in Ceilometer tagged with 
something that is unique to the [scaling group, member] and referencable 
in the template source.  For the case of a scaling group whose member type 
is nested stack, you could invent a way to implicitly pass such tagging 
down through all the intervening abstractions.  I was supposing the 
preferred solution would be for the template author to explicitly do this.

 ...

Thanks,
Mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [heat] health maintenance in autoscaling groups

2014-07-01 Thread Mike Spreitzer
In AWS, an autoscaling group includes health maintenance functionality --- 
both an ability to detect basic forms of failures and an ability to react 
properly to failures detected by itself or by a load balancer.  What is 
the thinking about how to get this functionality in OpenStack?  Since 
OpenStack's OS::Heat::AutoScalingGroup has a more general member type, 
what is the thinking about what failure detection means (and how it would 
be accomplished, communicated)?

I have not found design discussion of this; have I missed something?

I suppose the natural answer for OpenStack would be centered around 
webhooks.  An OpenStack scaling group (OS SG = OS::Heat::AutoScalingGroup 
or AWS::AutoScaling::AutoScalingGroup or OS::Heat::ResourceGroup or 
OS::Heat::InstanceGroup) could generate a webhook per member, with the 
meaning of the webhook being that the member has been detected as dead and 
should be deleted and removed from the group --- and a replacement member 
created if needed to respect the group's minimum size.  When the member is 
a Compute instance and Ceilometer exists, the OS SG could define a 
Ceilometer alarm for each member (by including these alarms in the 
template generated for the nested stack that is the SG), programmed to hit 
the member's deletion webhook when death is detected (I imagine there are 
a few ways to write a Ceilometer condition that detects instance death). 
When the member is a nested stack and Ceilometer exists, it could be the 
member stack's responsibility to include a Ceilometer alarm that detects 
the member stack's death and hit the member stack's deletion webhook. 
There is a small matter of how the author of the template used to create 
the member stack writes some template snippet that creates a Ceilometer 
alarm that is specific to a member stack that does not exist yet.  I 
suppose we could stipulate that if the member template includes a 
parameter with name member_name and type string then the OS OG takes 
care of supplying the correct value of that parameter; as illustrated in 
the asg_of_stacks.yaml of https://review.openstack.org/#/c/97366/ , a 
member template can use a template parameter to tag Ceilometer data for 
querying.  The URL of the member stack's deletion webhook could be passed 
to the member template via the same sort of convention.  When Ceilometer 
does not exist, it is less obvious to me what could usefully be done.  Are 
there any useful SG member types besides Compute instances and nested 
stacks?  Note that a nested stack could also pass its member deletion 
webhook to a load balancer (that is willing to accept such a thing, of 
course), so we get a lot of unity of mechanism between the case of 
detection by infrastructure vs. application level detection.

I am not entirely happy with the idea of a webhook per member.  If I 
understand correctly, generating webhooks is a somewhat expensive and 
problematic process.  What would be the alternative?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [heat] autoscaling across regions and availability zones

2014-07-01 Thread Mike Spreitzer
An AWS autoscaling group can span multiple availability zones in one 
region.  What is the thinking about how to get analogous functionality in 
OpenStack?

Warmup question: what is the thinking about how to get the levels of 
isolation seen between AWS regions when using OpenStack?  What is the 
thinking about how to get the level of isolation seen between AWS AZs in 
the same AWS Region when using OpenStack?  Do we use OpenStack Region and 
AZ, respectively?  Do we believe that OpenStack AZs can really be as 
independent as we want them (note that this is phrased to not assume we 
only want as much isolation as AWS provides --- they have had high profile 
outages due to lack of isolation between AZs in a region)?

I am going to assume that the answer to the question about ASG spanning 
involves spanning OpenStack regions and/or AZs.  In the case of spanning 
AZs, Heat has already got one critical piece: the OS::Heat::InstanceGroup 
and AWS::AutoScaling::AutoScalingGroup types of resources take a list of 
AZs as an optional parameter.  Presumably all four kinds of scaling group 
(i.e., also OS::Heat::AutoScalingGroup and OS::Heat::ResourceGroup) should 
have such a parameter.  We would need to change the code that generates 
the template for the nested stack that is the group, so that it spreads 
the members across the AZs in a way that is as balanced as is possible at 
the time.

Currently, a stack does not have an AZ.  That makes the case of an 
OS::Heat::AutoScalingGroup whose members are nested stacks interesting --- 
how does one of those nested stacks get into the right AZ?  And what does 
that mean, anyway?  The meaning would have to be left up to the template 
author.  But he needs something he can write in his member template to 
reference the desired AZ for the member stack.  I suppose we could 
stipulate that if the member template has a parameter named 
availability_zone and typed string then the scaling group takes care 
of providing the right value to that parameter.

To spread across regions adds two things.  First, all four kinds of 
scaling group would need the option to be given a list of regions instead 
of a list of AZs.  More likely, a list of contexts as defined in 
https://review.openstack.org/#/c/53313/ --- that would make this handle 
multi-cloud as well as multi-region.  The other thing this adds is a 
concern for context health.  It is not enough to ask Ceilometer to monitor 
member health --- in multi-region or multi-cloud you also have to worry 
about the possibility that Ceilometer itself goes away.  It would have to 
be the scaling group's responsibility to monitor for context health, and 
react properly to failure of a whole context.

Does this sound about right?  If so, I could draft a spec.

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [heat] One more lifecycle plug point - in scaling groups

2014-07-01 Thread Mike Spreitzer
Thinking about my favorite use case for lifecycle plug points for cloud 
providers (i.e., giving something a chance to make a holistic placement 
decision), it occurs to me that one more is needed: a scale-down plug 
point.  A plugin for this point has a distinctive job: to decide which 
group member(s) to remove from a scaling group (i.e., 
OS::Heat::AutoScalingGroup or OS::Heat::InstanceGroup or 
OS::Heat::ResourceGroup or AWS::AutoScaling::AutoScalingGroup).  The 
plugin's signature could be something like this: given a list of group 
members and a number to remove, return the list of members to remove (or, 
equivalently, return the list of members to keep).  What do you think?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


  1   2   3   >