Re: [Openstack-operators] Rescheduling neutron dhcp agents?

[Xav Paice]

Yeah, it's an odd one for sure, in my experience at least they do need to
be manually re-scheduled.

On Sun, 6 Aug 2017 at 11:27 Curtis  wrote:

> I'm in the process of testing out moving around some neutron services.
> I'm starting with dhcp-agent. I've got neutron set to have 3 dhcp
> instances per network.
>
> I set the admin state of a single dhcp agent to down, then all the
> dhcp namespaces were automatically removed from the node running dhcp
> agent. Then I deleted the agent thinking the dhcp services would get
> rescheduled to another node, but they were not. So now I've just got
> two dhcp servers for most networks.
>
> Am I, as admin, expected to now do something like add all the networks
> with only 2 dhcp servers to a new dhcp-agent (using neutron
> dhcp-agent-network-add) or should neutron reschedule them
> automagically?
>
> I'm hoping I just missed a step here. :)
>
> Thanks,
> Curtis.
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ubuntu package for Octavia

[Xav Paice]

I highly recommend looking in to Giftwrap for that, until there's UCA
packages.

The thing missing from the packages that Giftwrap will produce is init
scripts, config file examples, and the various user and directory setup
stuff.  That's easy enough to put into config management or a separate
package if you wanted to.

On 13 October 2016 at 01:25, Lutz Birkhahn  wrote:

> Has anyone seen Ubuntu packages for Octavia yet?
>
> We’re running Ubuntu 16.04 with Newton, but for whatever reason I can not
> find any Octavia package…
>
> So far I’ve only found in https://wiki.openstack.org/
> wiki/Neutron/LBaaS/HowToRun the following:
>
>  Ubuntu Packages Setup: Install octavia with your favorite
> distribution: “pip install octavia”
>
> That was not exactly what we would like to do in our production cloud…
>
> Thanks,
>
> /lutz
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-operators][ceph][nova] How do you handle Nova on Ceph?

[Xav Paice]

On Mon, 2016-10-10 at 13:29 +, Adam Kijak wrote:
> Hello,
> 
> We use a Ceph cluster for Nova (Glance and Cinder as well) and over
> time,
> more and more data is stored there. We can't keep the cluster so big
> because of 
> Ceph's limitations. Sooner or later it needs to be closed for adding
> new 
> instances, images and volumes. Not to mention it's a big failure
> domain.

I'm really keen to hear more about those limitations.


> 
> How do you handle this issue?
> What is your strategy to divide Ceph clusters between compute nodes?
> How do you solve VM snapshot placement and migration issues then
> (snapshots will be left on older Ceph)?

Having played with Ceph and compute on the same hosts, I'm a big fan of
separating them and having dedicated Ceph hosts, and dedicated compute
hosts.  That allows me a lot more flexibility with hardware
configuration and maintenance, easier troubleshooting for resource
contention, and also allows scaling at different rates.

> 
> We've been thinking about features like: dynamic Ceph configuration
> (not static like in nova.conf) in Nova, pinning instances to a Ceph
> cluster etc.
> What do you think about that?
> 
> 
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operato
> rs


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Rados Gateway to Swift migration

[Xav Paice]

On Wed, 2016-10-05 at 19:29 +, Fox, Kevin M wrote:
> Did you try it with jewel? If not, what version?
> 

>From Emperor up to Hammer, haven't upgraded since then.  I see that
there's a number of significant changes, some of the limitations we were
finding to be a problem may be gone now.

> Thanks,
> Kevin
> ________
> From: Xav Paice [xavpa...@gmail.com]
> Sent: Wednesday, October 05, 2016 12:12 PM
> To: George Mihaiescu
> Cc: OpenStack Operators
> Subject: Re: [Openstack-operators] Rados Gateway to Swift migration
> 
> On Wed, 2016-10-05 at 07:19 -0400, George Mihaiescu wrote:
> > Hi Xav,
> >
> > We are trying to get usage metrics for radosgw as well for internal cost 
> > recovery. Can you please share how far you got in that process and what it 
> > was missing?
> >
> 
> To be honest, we didn't get very far and that's one of the reasons for
> using Swift instead.  There were limitations with permissions that we
> found very difficult to get around, without having a data collection
> user added to each and every project.
> 
> > Thank you,
> > George
> >
> > > On Oct 5, 2016, at 1:57 AM, Xav Paice <xavpa...@gmail.com> wrote:
> > >
> > >> On Wed, 2016-10-05 at 15:42 +1100, Blair Bethwaite wrote:
> > >> Nice! But I'm curious, why the need to migrate?
> > >
> > > Hmm.  I want to be diplomatic since both are great for their thing.
> > >
> > > For us, the main reason was simply that we wanted replication of the
> > > object storage between regions (we started the process before that was a
> > > feature in RGW), but also being a public cloud we also wanted to be able
> > > to bill customers for their usage, and we were finding that incredibly
> > > difficult with Rados Gateway in comparison to Swift.
> > >
> > > That, and we found customers were using RGW as a backup, and that's on
> > > the same storage back end as our Cinder and Glance - moving to a
> > > different platform makes it separate.
> > >
> > > There's a few other features in Swift that aren't in RGW, and we have
> > > customers asking for them, which really matters a lot to us.
> > >
> > > There's pros and cons for both, I don't regret us using RGW but it just
> > > doesn't suit our needs right now.
> > >
> > >
> > > ___
> > > OpenStack-operators mailing list
> > > OpenStack-operators@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
> 
> 
> 
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators




___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Rados Gateway to Swift migration

[Xav Paice]

On Wed, 2016-10-05 at 07:19 -0400, George Mihaiescu wrote:
> Hi Xav,
> 
> We are trying to get usage metrics for radosgw as well for internal cost 
> recovery. Can you please share how far you got in that process and what it 
> was missing?
> 

To be honest, we didn't get very far and that's one of the reasons for
using Swift instead.  There were limitations with permissions that we
found very difficult to get around, without having a data collection
user added to each and every project.

> Thank you,
> George
> 
> > On Oct 5, 2016, at 1:57 AM, Xav Paice <xavpa...@gmail.com> wrote:
> > 
> >> On Wed, 2016-10-05 at 15:42 +1100, Blair Bethwaite wrote:
> >> Nice! But I'm curious, why the need to migrate?
> > 
> > Hmm.  I want to be diplomatic since both are great for their thing.
> > 
> > For us, the main reason was simply that we wanted replication of the
> > object storage between regions (we started the process before that was a
> > feature in RGW), but also being a public cloud we also wanted to be able
> > to bill customers for their usage, and we were finding that incredibly
> > difficult with Rados Gateway in comparison to Swift.
> > 
> > That, and we found customers were using RGW as a backup, and that's on
> > the same storage back end as our Cinder and Glance - moving to a
> > different platform makes it separate.
> > 
> > There's a few other features in Swift that aren't in RGW, and we have
> > customers asking for them, which really matters a lot to us.
> > 
> > There's pros and cons for both, I don't regret us using RGW but it just
> > doesn't suit our needs right now.
> > 
> > 
> > ___
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators




___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Rados Gateway to Swift migration

[Xav Paice]

On Wed, 2016-10-05 at 15:42 +1100, Blair Bethwaite wrote:
> Nice! But I'm curious, why the need to migrate?

Hmm.  I want to be diplomatic since both are great for their thing.

For us, the main reason was simply that we wanted replication of the
object storage between regions (we started the process before that was a
feature in RGW), but also being a public cloud we also wanted to be able
to bill customers for their usage, and we were finding that incredibly
difficult with Rados Gateway in comparison to Swift.

That, and we found customers were using RGW as a backup, and that's on
the same storage back end as our Cinder and Glance - moving to a
different platform makes it separate.

There's a few other features in Swift that aren't in RGW, and we have
customers asking for them, which really matters a lot to us.

There's pros and cons for both, I don't regret us using RGW but it just
doesn't suit our needs right now.


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Rados Gateway to Swift migration

[Xav Paice]

On Tue, 2016-10-04 at 17:48 -0600, Curtis wrote:
> Maybe you have someone on staff who loves writing lua (for haproxy)? :)
> 

Well, maybe not that far, but yeah we're now thinking down that route.
If we get there, I'll quickly write something up about it.  Many thanks
for the suggestion :)


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Rados Gateway to Swift migration

[Xav Paice]

On Tue, 2016-10-04 at 21:56 +, Fox, Kevin M wrote:
> OpenStack really needs a way to have a control api for selecting a swift 
> "flavor", and letting you have multiple swift endpoints within, so swift the 
> software, radowgw, and vendor endpoints can all co'exist.
> 

Just thinking about it, we use haproxy and the url for requests does
include the project id, so we might be able to regex the requests and
direct to the appropriate backend.  It could make for complex support
issues in the meantime, but a much smoother migration.

> Kevin
> ________
> From: Xav Paice [xavpa...@gmail.com]
> Sent: Tuesday, October 04, 2016 2:45 PM
> To: Curtis
> Cc: OpenStack Operators
> Subject: Re: [Openstack-operators] Rados Gateway to Swift migration
> 
> On Tue, 2016-10-04 at 07:18 -0600, Curtis wrote:
> > On Mon, Oct 3, 2016 at 3:31 PM, Xav Paice <xavpa...@gmail.com> wrote:
> > > Hi,
> > >
> > > We're in the process of migrating our object storage from Rados Gateway
> > > to Swift, and I was wondering if anyone has experiences with the data
> > > migration steps from one to the other?
> > >
> > > In particular, we're currently copying each object and container one by
> > > one, but the process of inspecting each and every object is incredibly
> > > slow.  The logic we have is kind of rsync-like, so we can repeat and
> > > iterate until it's close, but the final go-live cutover will still take
> > > many hours to complete.  Ideas on how to get over that would be very
> > > much appreciated!
> >
> > Could you load balance to on backend or another based on whether or
> > not the account has been migrated yet? I haven't thought that through
> > completely...
> >
> 
> That would be the preferable situation, migrating one customer at a
> time, but we couldn't figure out how to achieve that goal without
> writing some kind of middleware/proxy type of thing, or being clever
> with haproxy.  We may yet have to go down that route, but I'm hoping
> not!
> 
> 
> > Thanks,
> > Curtis.
> >
> > >
> > > Many thanks
> > >
> > >
> > > ___
> > > OpenStack-operators mailing list
> > > OpenStack-operators@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
> >
> >
> 
> 
> 
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Rados Gateway to Swift migration

[Xav Paice]

On Tue, 2016-10-04 at 07:18 -0600, Curtis wrote:
> On Mon, Oct 3, 2016 at 3:31 PM, Xav Paice <xavpa...@gmail.com> wrote:
> > Hi,
> >
> > We're in the process of migrating our object storage from Rados Gateway
> > to Swift, and I was wondering if anyone has experiences with the data
> > migration steps from one to the other?
> >
> > In particular, we're currently copying each object and container one by
> > one, but the process of inspecting each and every object is incredibly
> > slow.  The logic we have is kind of rsync-like, so we can repeat and
> > iterate until it's close, but the final go-live cutover will still take
> > many hours to complete.  Ideas on how to get over that would be very
> > much appreciated!
> 
> Could you load balance to on backend or another based on whether or
> not the account has been migrated yet? I haven't thought that through
> completely...
> 

That would be the preferable situation, migrating one customer at a
time, but we couldn't figure out how to achieve that goal without
writing some kind of middleware/proxy type of thing, or being clever
with haproxy.  We may yet have to go down that route, but I'm hoping
not!


> Thanks,
> Curtis.
> 
> >
> > Many thanks
> >
> >
> > ___
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
> 
> 



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Kilo ->-> Mitaka anyone have notes :)

[Xav Paice]

FWIW, I did that with Cinder and it was great except for one DB change we
had to make regarding the server name for volumes -
http://dischord.org/2015/12/22/cinder-multi-backend-with-multiple-ceph-pools/
is a pretty good description of what I saw.  I suspect that was just a poor
configuration from our Kilo setup rather than a change specifically related
to the upgrade, but the upgrade brought it to light.

Haven't yet taken the same step with others, we're doing the upgrades one
thing at a time and for some (Nova) we'll be jumping through Liberty
because of the risk of jumping two steps at a time.

On 25 June 2016 at 06:16, Jonathan Proulx  wrote:

> Hi All,
>
> I about to start testing for our Kilo->Mitaka migration.
>
> I seem to recall many (well a few at least) people who were looking to
> do a direct Kilo to Mitaka upgrade (skipping Liberty).
>
> Blue Box apparently just did and I read Stefano's blog[1] about it,
> and while it gives me hope my plan is possible it's not realy a
> technical piece.
>
> I'm on my 7th version of OpenStack for this cloud now so not my first
> redeo as they say, but other than read Liberty and Mitaka release
> notes carefully and test like crazy wonder if anyone has seen specific
> issues or has specific advice for skiping a step here?
>
> Thanks,
> -Jon
>
> --
> [1]
> https://www.dreamhost.com/blog/2016/06/21/dreamcompute-goes-m-for-mitaka-without-unleashing-the-dragons/
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Packaging Virtualenvs

[Xav Paice]

Can I suggest that using the tool https://github.com/openstack/giftwrap
might make live a bunch easier?

I went down a similar path with building Debs in a venv using
dh_virtualenv, with some good success when I sorted the shebang. I later
found that the debs produced by Giftwrap are not only very easy to build
and test, but also take a bunch less effort to maintain and create new
packages for new things.  To run the resulting code, I just symlink the
${venv}/bin/$binary to /usr/local/bin and run the thing using very
similar init scripts to the ones supplied by the distro packages.  Works
like a charm, because the shebang in the binary points at the venv, not
the system python.

I do, however, package the init scripts, sample configs, etc in a
separate .deb, which is really very quick and easy and allows me to
control the bits I want to, and let Giftwrap take care of the OpenStack
code repos.


On Thu, 2016-06-23 at 23:40 +, Matt Joyce wrote:
> I want the script to dynamically instantiate the venv is call activate
> this at execution time and deactivate when done.
> 
> 
> 
> On June 23, 2016 5:12:07 PM EDT, Doug Hellmann 
> wrote:
> Excerpts from Silence Dogood's message of 2016-06-23 15:45:34 -0400:
>  I know from conversations that a few folks package their 
> python apps as
>  distributable virtualenvs.   spotify created dh-virtualenv 
> for this.  you
>  can do it pretty simply by hand.
>  
>  I built a toolchain for building rpms as distributable 
> virtualenvs and that
>  works really well.
>  
>  What I'd like to do is make it so that every app that's 
> built as a
>  virtualenv gets setup to automatically execute at call time 
> in their
>  virtualenv.
>  
>  I see two options:
>  
>  1)  Dynamically generate a wrapper script during build and 
> put it in the
>  RPM.  Call the wrapper.
>  
>  2)  Created a dependency global module ( as an rpm ) set it 
> as a
>  dependency.  And basically it'd be an autoexecuting import 
> that
> 
> instantiates the virtualenv.  it would probably know all it 
> needs to
>  because I am building all my packages to an internal 
> standard.  Then when
>  building the APP rpm all I need to do is inject an import 
> into the import
>  chain if it's being built as a virtualenv.  Then I have what 
> are
>  effectively statically compiled python apps.
>  
>  I like 2.  But 1 isn't very bad.  Both are a little hokey.
>  
>  Was curious if folks might have a preference, or a better 
> idea.
>  
>  Thanks.
>  
>  Matt
> 
> I'm not sure what you mean by a "wrapper script".  If you run the
> Python console script from within the virtualenv you've packaged,
> you shouldn't need to do anything to "activate" that environment
> separately because it should have the correct shebang line.
> 
> Are you seeing different behavior?
> 
> Doug
> 
> 
> __
> 
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] VPNaaS and FWaaS

[Xav Paice]

On 3 May 2016 at 05:03, Matt Jarvis  wrote:

> Thanks for the clarification Kyle.
>
> On 2 May 2016 at 14:33, Kyle Mestery  wrote:
>
>> On Fri, Apr 29, 2016 at 8:01 AM, Matt Jarvis
>>  wrote:
>> > I know there are operators relying on these functions, particularly in
>> the
>> > public cloud space in Europe, so this would impact those people.
>> >
>> I'm actually really surprised that people are *using* FWaaS. It's been
>> marked experimental for over 3 years now, and it only recently in
>> Liberty received work which made it somewhat useful, which was the
>> ability to apply a firewall on a specific Neutron router rather than
>> all tenant routers. FWaaS in production sounds pretty risky to me, but
>> I supposed that our fault for not being clear on it's readiness.
>>
>>
It might be good at this stage to differentiate between the number of
people using FWaaS and VPNaaS.  It might be that the FWaaS is much less
used than VPN, and while we've had a large number of support calls
regarding VPNaaS, using the service has meant that we can operate as a
public cloud despite having a very limited amount of IPv4 address space.
Without VPNaaS, we would have to make some very difficult changes to our
operations and probably wind up pouring resources into maintaining
something that doesn't provide such a nice customer experience.  We've not
yet worked out what FWaaS is for, and our customers haven't asked us for it.



> > If we have metrics that a constituent part of the user community need
>> these
>> > functions, then we can try and find a way to help the Neutron team to
>> cover
>> > the resourcing gaps.
>> >
>> If people are using these, IMHO that's another reason to keep them
>> around. I've already said that we have at least one large user of VPN,
>> so that project will continue to be worked on even if it's removed
>> from Neutron.
>>
>
I would expect large users of a project to be able to contribute at least
_some_ resources to keep the code alive.  As a small user of VPNaaS , I
would also expect to contribute some resources - but we're too small to be
a significant contributor here.

I'm not sure how OSIC would relate, particularly as this is low/absent in
their priorities, but if the only barrier to people working on VPNaaS is
getting a test/dev cluster to work with then surely it's a barrier that can
be removed.  I would expect the developer time to be the biggest hurdle.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Cloud Upgrade Strategies

[Xav Paice]

On 10 March 2016 at 19:26, Yuriy Brodskiy  wrote:

> building a new cloud is not practical for real production environments.
> even if you can afford it, how do you migrate data?
>
> We have been doing upgrades for a while now, and came up with few basic
> principles:
> 1) you don't have to upgrade all at the same time. do it component at the
> time
> 2) stand up a new version along side of an existing one, test it and then
> flip DNS
>
> Take a look at presentation team did during Vancouver summit
>
> https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/10-minutes-openstack-upgrades-done
>
>
(replying to the list this time, and regretting using gmail)

I readily admit to not having watched that video (but will!) - one
question.  How do you deal with the db migration if you have two versions
running at the same time?

>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Service Catalog TNG urls

[Xav Paice]

On 7 December 2015 at 14:33, Clint Byrum  wrote:

>
> Indeed, this is one paradigm where it actually creates a hardened core,
> not a squisy one, so I think it's not a terrible idea.
>
>
I tend to agree, just wish we were already there :)


>
>
> What I mean is, you'd just have two completely separate _services_,
> instead of one service, with two separate endpoint "interface" entries:
>
> 
>

I guess if the clients already have mechanisms to determine whether to use
the public url or the admin url, they can determine which service to look
up just the same.  However, that brings me back to wondering if there's any
real difference?   I'm missing something.  But like you say, if we don't
have Keystone running such that we trust it enough to call it hardened,
then that's a much more pressing issue.

What are other public cloud ops doing wrt public facing API services that
can expose admin type operations?
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [ceilometer][keystone][billing] RBAC restrictions of Ceilometer's Event API prevents billing of Openstack cloud

[Xav Paice]

I don't fully understand the architecture of our rating project, but as I
understand it we use a service with rights to read ceilometer info (in our
case, pollsters and notifications) and then aggregate that into another
database which we query to get the actual billing information.   We bill
monthly, and getting a month's worth of data for each tenant in one hit
would crush our older, Mongo based, Ceilometer into dust.  The tiny
database that contains the distilled information is quick and easy to
query, to the point where we can display that information to customers as
an extra panel in the dashboard.

Just a side note - what would happen if you missed an event?  Say, e.g.,
rabbit had a bad day... or something just wasn't running... We found
pollsters gave us the assurance to mean we'd only lose up to 10 mins of
data, which is generally OK when we bill by the hour.

On 7 December 2015 at 23:01, Christian Brinker  wrote:

> Hi,
>
> my company is currently starting to implement a public Openstack
> cloud. I am part of the developer team creating a billing system
> towards our customers. We want to use
> Ceilometer's Event API (Liberty release) to retreive the usage
> information (as /v2/events) of our customers projects(aka tenants).
> Unfortunately, the RBAC filter
> prevents REST calls towards the /v2-Web-API from users who are not
> member of the project (or are their admin). But adding a user to all
> projects with a distinc
> ceilometer-reader role or admin role seems not fourtunate to us
> because to want to serve admin role users to their own domain to each
> customer. So the ceilometer-reader
> user could be removed by a customer. Due to this, we ran into some
> kind of deadlock of good solutions and would be happy to get any help:
>
> - Is there another/common way to retrieve the event based usage
> information in a way to generate billing information? For example
> volume A was created at t1 and deleted
> at t2.
> - Is there a way to get a project scope token from keystone through
> some kind of cloud admin user which is not part of the project?
> - Is there a way to change Ceilometers policy.json in a way to
> retrieve data from all projects with a admin on the default project or
> someone similiar?
>
> Thanks for your efforts.
>
> Greetings,
> Christian Brinker
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Service Catalog TNG urls

[Xav Paice]

On 7 December 2015 at 05:38, Clint Byrum  wrote:

> Excerpts from Xav Paice's message of 2015-12-05 13:26:23 -0800:
> > >
>
> I respect that this is what works for you and we shouldn't require you to
> change your ways without good reason. However, I just want to point out
> that if you don't trust Keystone's own ACL's to prevent administrative
> access by users who haven't been granted access, then you also don't
> trust Keystone to keep users out of each-others accounts!
>
>
That's an excellent point, and one which scares me quite a lot.  But that's
the sad reason we need two lots of API servers - so even if someone were to
get hold of an admin userid/password, they still can't go deleting the
entire cloud.  It does at least limit the damage.



> That said, if there really is a desire to keep admin functions separate
> from user functions, why not formalize that and make it an entirely
> separate service in the catalog? So far, Keystone is the only service
> to make use of "adminurl". So a valid path forward is to simply make it
> a different entry.
>

Keystone is indeed the only one that does this - I hesitate to say "right"
because it might not be.

I'm not sure I follow when you say separate service - you mean a completely
different service, with a full set of endpoints?  Makes sense if the
projects that use the catalogue also honour that, but I don't know I see
the difference between having a different service for admin requests, and a
split admin url and public url.  Maybe I'm just being thick here, but I had
thought that was the original intention despite it never being used by
anyone other than Keystone.



>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Neutron getting stuck creating namespaces

[Xav Paice]

Hi,

Over the last few months we've had a few incidents where the process to
create network namespaces (Neutron, OVS) on the network nodes gets 'stuck'
and prevents not only the router it's trying to create from finishing, but
all further namespace operations too.

This has usually finished up with either us rebooting the node pretty fast
afterwards, or the node rebooting itself.

It looks very much like we're affected by
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1403152 but the notes
say it's fixed in the kernel we're running.  I've asked the clever person
who checked it to make some extra notes in the bug report.

It looks very much like when we have a bunch of load on the box the thing
is more likely to trigger - I was wondering if other ops have a max ratio
of routers per network node?  I would have thought our current max of 150
routers per node would be pretty light, but with the dhcp namespaces as
well that's ~450 namespaces on a box and maybe that's an issue?

Thanks
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Running mixed stuff Juno & Kilo , Was: cinder-api with rbd driver ignores ceph.conf

[Xav Paice]

We're running mixed Juno and Kilo, and upgrading one component at a time.
Two issues with that so far:
1. The puppet modules don't play nice together (Juno/Kilo) - there was some
serious re-factor done between those releases and they don't work at all
together.  I hacked the Juno modules to write Kilo configs till I can
upgrade the Puppet modules all together in one hit, after upgrading
Openstack.
2. The distro packages need lots of libraries that don't play between
versions.  I re-packaged everything with compatible package names for the
Puppet modules, and in a virtual environment.  Take a look at Giftwrap if
you want to do that easily.

On 19 November 2015 at 02:02, Arne Wiebalck  wrote:

> We do run Cinder on dedicated controllers, though, so no mix of Kilo and
> Juno services on
> the controllers.
>
> Cheers,
>  Arne
>
>
> On 18 Nov 2015, at 13:19, Belmiro Moreira <
> moreira.belmiro.email.li...@gmail.com> wrote:
>
> Hi Saverio,
> we always upgrade one component at a time.
> Cinder was one of the first components that we upgraded to kilo,
> meaning that other components (glance, nova, ...) were running Juno.
>
> We didn't have any problem with this setup.
>
> Belmiro
> CERN
>
> On Tue, Nov 17, 2015 at 6:01 PM, Saverio Proto  wrote:
>
>> Hello there,
>>
>> I need to quickly find a workaround to be able to use ceph object map
>> features for cinder volumes with rbd backend.
>>
>> However, upgrading everything from Juno to Kilo will require a lot of
>> time for testing and updating all my puppet modules.
>>
>> Do you think it is feasible to start updating just cinder to Kilo ?
>> Will it work with the rest of the Juno components ?
>>
>> Has someone here experience in running mixed components between Juno and
>> Kilo ?
>>
>> thanks
>>
>> Saverio
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Fwd: OPs Midcycle location discussion.

[Xav Paice]

For us NZ (and maybe Aus) folk to attend, getting to Europe is (literally)
twice the distance of US, but Asia is about the same (language aside).I
find the Summit really valuable in that a large and diverse group get
together and the discussion is live, and in the same time zone - I'd be sad
to lose that focus if we split the mid cycle, but if it was regional I'd be
more likely to justify the cost of getting there.  A set of air fares and
accommodation for the Summit every 6 months is hard enough to fund,
doubling it isn't easy.

Maybe it would be good to re-state the purpose/goal of the mid cycle and
see which option matches that better?

On 17 November 2015 at 06:37, Matt Fischer  wrote:

> I think that sticking with a singular official one is the plan. It's
> difficult enough for the foundation to line up sponsors/hosts etc for a
> single meet-up. I also think that there are some US/Asia folks that will
> attend a midcycle in Europe and by also hosting a competing one locally you
> may reduce the attendance at the main one which defeats the purpose. Those
> midcycles work best when we have lots of different voices providing input.
>
> On Mon, Nov 16, 2015 at 10:06 AM, Jonathan Proulx 
> wrote:
>
>> On Mon, Nov 16, 2015 at 04:55:33PM +, Kruithof, Piet wrote:
>> :Sorry, late to the conversation and maybe missing a bit of context.
>> :
>> :How may regional meetings are we thinking?  2-3? Or more?
>>
>> My basic question was One or Many.
>>
>> If Many then that's a further question, but probably 3 (north america,
>> asia, europe)  or possibly 4 (+ south america)
>>
>> :
>> :Piet
>> :
>> :
>> :
>> :
>> :Piet Kruithof
>> :Sr UX Architect, HP Helion Cloud
>> :PTL, OpenStack UX project
>> :
>> :"For every complex problem, there is a solution that is simple, neat and
>> wrong.”
>> :
>> :H L Menken
>> :
>> :
>> :From: Matt Jarvis  matt.jar...@datacentred.co.uk>>
>> :Date: Monday, November 16, 2015 at 9:23 AM
>> :To: Jonathan Proulx >
>> :Cc: "openstack-operators@lists.openstack.org> openstack-operators@lists.openstack.org>" <
>> openstack-operators@lists.openstack.org> openstack-operators@lists.openstack.org>>
>> :Subject: Re: [Openstack-operators] OPs Midcycle location discussion.
>> :
>> :+1 from me, although I am admittedly biased ;) Personally I think the
>> wider participation in the ops feedback loop can only be a positive thing,
>> and there are definitely different perspectives and concerns to be had from
>> European operators given the different commercial landscape. I'm sure the
>> same is also true for Asia.
>> :
>> :On 16 November 2015 at 15:50, Jonathan Proulx  j...@csail.mit.edu>> wrote:
>> :Hi All,
>> :
>> :1st User Committee IRC meeting will be today at 19:00UTC on
>> :#openstack-meeting, we haven't exactly settled on an agenda yet but I
>> :hope to raise this issue the...
>> :
>> :It has been suggested that we make the February 15-16 European Ops
>> :Meetup in Manchester UK [1] the 'official' OPs Midcycle.  Previously
>> :all mid cycles have been US based.
>> :
>> :Personally I like the idea of broadening or geographic reach rather
>> :than staying concentrated in North America. I particularly like it
>> :being 'opposite' the summit location.
>> :
>> :This would likely trade off some depth of participation as fewer
>> :of the same people would be able to travel to all midcycles in person.
>> :
>> :Discuss...(also come by  #openstack-meeting at 19:00 UTC if you think
>> :this needs real time discussion)
>> :
>> :-Jon
>> :
>> :
>> :--
>> :
>> :1.
>> http://www.eventbrite.com/e/european-openstack-operators-meetup-tickets-19405855436?aff=es2
>> :
>> :___
>> :OpenStack-operators mailing list
>> :OpenStack-operators@lists.openstack.org> OpenStack-operators@lists.openstack.org>
>> :http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> :
>> :
>> :
>> :--
>> :Matt Jarvis
>> :Head of Cloud Computing
>> :DataCentred
>> :Office: (+44)0161 8703985
>> :Mobile: (+44)07983 725372
>> :Email: matt.jar...@datacentred.co.uk> matt.jar...@datacentred.co.uk>
>> :Website: http://www.datacentred.co.uk
>> :
>> :DataCentred Limited registered in England and Wales no. 05611763
>>
>> --
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [puppet] Can we install OpenStack components using puppet but through source?

[Xav Paice]

It's possible to do this if you build your own packages and name them the
same as the distro packages - then make sure that you have all the deps etc
as well.

I'm making packages for Ubuntu by re-using some of the Ubuntu Cloud
Archive, some of Debian, and some of my own packaging code, and building
packages now in a venv.  I then put those packages in an apt repo, add the
repo to the box, and the Puppet modules pick up the custom packages just
fine.

I guess it's down to your goals though - if you want to build from source
to get fixes in fast, then this works, but if you're after a developer
workstation then you probably want to use Devstack.  It's the same code,
just miles easier to develop with in that format because all the git stuff
is taken care of.

BTW, +1 for openstack-ansible, and I'll add a note to go find out about
Giftwrap too (https://github.com/blueboxgroup/giftwrap) - you can fix that
to use your own repo if you want.

For my next iteration I'll have a go at Giftwrap for the actual Python
code, and make sundry packages to keep Puppet happy and plug in the distro
specific stuff like init scripts etc that are missing from the sources.

On 13 November 2015 at 21:06, George Paraskevas 
wrote:

> Hello,
>
> I believe you can do so with openstack-ansible, its a set of ansible
> playbooks that orchestrate openstack from source using wheels. I think
> there you can select different branches for selected components.
> If this is what you want
> https://github.com/openstack/openstack-ansible
>
> Thanks
> George.
>
> On Fri, 13 Nov 2015, 09:07 Okan bhan  wrote:
>
>> Hi Guys,
>>
>> I'm trying installing OpenStack using puppet modules. I'm able to setup
>> all-in-one installation for liberty release.
>>
>> This works fine but I was hoping, if there is a way I can install
>> services (say keystone) from source code. Essentially I was hoping if I can
>> use puppet to setup a developer workstation.
>>
>> Thanks
>> Alok Kumar
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Kilo upgrade challenges

[Xav Paice]

Follow up to this...

It turns out that using a self signed SSL cert is problematic when python
requests bundles it's cacerts separately to the system list of ca certs (in
a virtual env).

This was solved by cat'ing the ca cert for our ca
>> /usr/share/python/python-heat/lib/python2.7/site-packages/requests/cacert.pem

It was bound to be something daft like that!

Many thanks for the replies, very much appreciated.


On 12 November 2015 at 13:20, Matt Kassawara <mkassaw...@gmail.com> wrote:

> Did you change anything under [keystone_authtoken] in the heat.conf file?
>
> On Wed, Nov 11, 2015 at 4:43 PM, Leslie-Alexandre DENIS <
> cont...@ladenis.fr> wrote:
>
>> Le 11/11/2015 05:46, Xav Paice a écrit :
>>
>> Hi,
>>
>> Late to the party, I'm only just doing the Kilo upgrade now (with a
>> couple of projects going direct to Liberty).  I seem to have hit a bit of a
>> snag, and I've now spent a bit too long banging my head against this, was
>> wondering if anyone else has advice/experiences to share.
>>
>> If it's a "you Muppet, you did X wrong" thing, I'd love to hear about it
>> - I'm 99.9% sure I've stuffed up a config somewhere.
>>
>> In short, after upgrading, say, Heat, to Kilo, and running the db
>> migration, restarting etc, the CLI is returning 'Authentication required'.
>> My user is admin, and nothing has changed that I'm aware of.  I can't see
>> anything particularly new in the logs for keystone, nor in heat, except
>> that I now see "WARNING keystonemiddleware.auth_token [-] Authorization
>> failed for token".  I'm not sure if that's a problem or not though.
>>
>> Some details etc are in http://paste.openstack.org/show/478501/ -> from
>> a dev environment so not even sanitized.
>>
>> Anyone been there?
>>
>> Thanks
>> Xav
>>
>>
>> ___
>> OpenStack-operators mailing 
>> listOpenStack-operators@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>> Hello Xav,
>>
>> I faced similar problems last few weeks, during an Icehouse to Kilo
>> upgrade, regarding to Cinder and I found out that it was due to client
>> version.
>> You can find the correct version in
>> https://github.com/openstack/heat/blob/stable/kilo/requirements.txt
>> At least, you can double check it and eventually solve this.
>>
>> My 2 cents,
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Kilo upgrade challenges

[Xav Paice]

Hi,

Late to the party, I'm only just doing the Kilo upgrade now (with a couple
of projects going direct to Liberty).  I seem to have hit a bit of a snag,
and I've now spent a bit too long banging my head against this, was
wondering if anyone else has advice/experiences to share.

If it's a "you Muppet, you did X wrong" thing, I'd love to hear about it -
I'm 99.9% sure I've stuffed up a config somewhere.

In short, after upgrading, say, Heat, to Kilo, and running the db
migration, restarting etc, the CLI is returning 'Authentication required'.
My user is admin, and nothing has changed that I'm aware of.  I can't see
anything particularly new in the logs for keystone, nor in heat, except
that I now see "WARNING keystonemiddleware.auth_token [-] Authorization
failed for token".  I'm not sure if that's a problem or not though.

Some details etc are in http://paste.openstack.org/show/478501/ -> from a
dev environment so not even sanitized.

Anyone been there?

Thanks
Xav
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [neutron] Any users of Neutron's VPN advanced service?

[Xav Paice]

On 06/08/15 07:56, Kyle Mestery wrote:
 Operators:

 We (myself, Paul and Doug) are looking to better understand who might
 be using Neutron's VPNaaS code. We're looking for what version you're
 using, how long you're using it, and if you plan to continue deploying
 it with future upgrades. Any information operators can provide here
 would be fantastic!

We're running it since Icehouse, and there's one or two issues which are
known bugs with upstream fixes in progress, but overall we're happy with
it.  It's miles easier for our customers to drive than VPN inside VMs,
and the ease helps us retain our only-too-scarce IPv4 space.  Our
customers would be very upset if we discontinued use.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Dynamic Policy

[Xav Paice]

On 06/08/15 04:01, Kris G. Lindgren wrote:
 We ran into this as well.

 What we did is create an external to keystone api, that we expose to our
 end users via a UI.  The api will let user create projects (with a
 specific defined quota) and also add users with the project admins  role
 to the project.  Those admins can add/remove users from the project and
 also delete the project.  You can also be a member, where you have the
 ability to spin up vm's under the project but not add/remove users or
 remove the project.  We also do some other stuff to clean up items in a
 project before its deleted.  We are working to move this functionality out
 of the current external API and into keystone.  I believe we were going to
 look at waffle-haus to add a paste filter to intercept the project create
 calls and do the needful.
We're working on something similar, but haven't rolled it to production
yet.  Is your code available open-source somewhere?  Ours will be once
it's clean-ish and tested properly, but not yet lest we lead someone
into pain and misery.

One of the goals you didn't mention above, but I'm sure you also noted,
was that changing passwords or setting an initial password isn't exactly
clear - we're working on getting a one time link set that an initial
user can be sent to be able to set their own first password.

 We also modified the policy.json files for the services that we care about
 to add the new roles that we created.

Not the easiest task to either get right, or make sure that the files
are distributed around in an HA setting.  But absolutely necessary.

 
  
 Kris Lindgren
 Senior Linux Systems Engineer
 GoDaddy, LLC.




___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Problem with migration from Havana to IceHouse

[Xav Paice]

On 21/01/15 23:43, Alvise Dorigo wrote:
 Hi,
 I've an Havana IaaS composed by:

 1 controller node
 1 network node
 1 compute node

 and using Neutron as networking, and rabbitmq as AMQP.

 The network node runs the agents (dhcp, l3, openvswitch, metadata).
 The controller runs Keystone, glance, Nova APIs, Neutron server.

 The compute node runs Nova Compute.

 I've followed this guide:
 http://docs.openstack.org/openstack-ops/content/upgrades_havana-icehouse-rhel.html
 to perform a migration from Havana to IceHouse.

 After migration (without any apparent error) I get a lot of errors for
 the several services related to the AMQP.

 For example in the neutron server's log I see:

 http://pastebin.com/2pSpP4mu

 In the nova api's log I see:

 http://pastebin.com/uCzyhxb6

 and a similar one for Nova conductor:

 http://pastebin.com/3B0jvT6i

 Did I miss something ? I didn't see anything directly related to AMQP
 in the guide cited abobe.


connection reset by peer is probably the thing to check on first -
looks pretty much like the rabbitmq server either isn't running, or
isn't configured correctly in the various OpenStack config files - did
you compare the old/new configs?  Maybe ensure you can telnet to the
rabbit port, and eliminate the network as a problem first.

 thanks,

 Alvise


 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators