Re: [Openstack-operators] VM monitoring suggestions

2016-11-17 Thread Joe Topjian
We have some custom scripts that run on the hypervisors which poll: virsh dominfo virsh domiflist etc The memory stats with "virsh dommemstat" are, AFAIK, not accurate since there's nothing triggering kvm / the vm to release unused memory. But all other virsh stuff works well for us. We don't re

[Openstack-operators] Encrypted Cinder Volume Deployment

2017-01-23 Thread Joe Topjian
Hi all, I'm investigating the options for configuring Cinder with encrypted volumes and have a few questions. The Cinder environment is currently running Kilo which will be upgraded to something between M-O later this year. The Kilo release supports the fixed_key setting. I see fixed_key is still

Re: [Openstack-operators] Encrypted Cinder Volume Deployment

2017-01-23 Thread Joe Topjian
the badly hashed key/password vs’s > what was configured. > > > > > > ___ > > Kris Lindgren > > Senior Linux Systems Engineer > > GoDaddy > > > > *From: *Joe Topjian > *Date: *Monday, Ja

Re: [Openstack-operators] Encrypted Cinder Volume Deployment

2017-02-05 Thread Joe Topjian
archives and has an update, do post :) Thanks, Joe On Mon, Jan 23, 2017 at 8:58 PM, Joe Topjian wrote: > Hi Kris, > > I came across that as well and I believe it has been fixed and ensures > existing volumes are accessible: > > https://github.com/openstack/nova/blob/8c3f7

Re: [Openstack-operators] libvirt freezing when loading Nova instance nwfilters

2017-02-22 Thread Joe Topjian
We ran into the "virsh nwfilter-list hanging indefinitely" thing back in early January. I spent hours and I almost went insane trying to figure it out. We weren't upgrading nodes, though, it just sort of happened. I have no idea if the following was the correct way of handling this, but this ultim

Re: [Openstack-operators] Flavors

2017-03-15 Thread Joe Topjian
Another benefit of flavors is that they provide ease of use. While there are users who are confident enough to spec out each instance they launch, I work with a lot of users who would feel overwhelmed if they had to do this. Providing a set of recommended instance specs can go a long way to lowerin

Re: [Openstack-operators] Flavors

2017-03-15 Thread Joe Topjian
e bundled. :) On Wed, Mar 15, 2017 at 10:31 PM, Joe Topjian wrote: > Another benefit of flavors is that they provide ease of use. While there > are users who are confident enough to spec out each instance they launch, I > work with a lot of users who would feel overwhelmed if they h

Re: [Openstack-operators] [neutron] Modify Default Quotas

2017-03-23 Thread Joe Topjian
We run a similar kind of script. I think in most cases, a Floating IP means a publicly routable IP, and those are now scarce resources. Because of that, I agree with what's been mentioned about a conservative floating IP quota. Since the other resource types aren't restricted by external availabi

Re: [Openstack-operators] backup to object store - tool recommendations

2017-03-27 Thread Joe Topjian
We use rclone quite a bit. It works great and has a wealth of features: http://rclone.org/ On Mon, Mar 27, 2017 at 7:50 AM, Nick Jones wrote: > On 27 Mar 2017, at 12:59, Marcus Furlong wrote: > > > On 27 March 2017 at 22:39, Blair Bethwaite > wrote: > > Hi all, > > Does anyone have any recom

Re: [Openstack-operators] [scientific] Resource reservation requirements (Blazar) - Forum session

2017-04-01 Thread Joe Topjian
On Sat, Apr 1, 2017 at 5:21 PM, Matt Riedemann wrote: > On 4/1/2017 8:36 AM, Blair Bethwaite wrote: > >> Hi all, >> >> The below was suggested for a Forum session but we don't yet have a >> submission or name to chair/moderate. I, for one, would certainly be >> interested in providing input. Do w

Re: [Openstack-operators] [scientific] Resource reservation requirements (Blazar) - Forum session

2017-04-03 Thread Joe Topjian
On Mon, Apr 3, 2017 at 8:20 AM, Jay Pipes wrote: > On 04/01/2017 08:32 PM, Joe Topjian wrote: > >> On Sat, Apr 1, 2017 at 5:21 PM, Matt Riedemann > <mailto:mriede...@gmail.com>> wrote: >> >> On 4/1/2017 8:36 AM, Blair Bethwaite wrote: >> >

[Openstack-operators] Successful nova-network to Neutron Migration

2017-05-20 Thread Joe Topjian
Hi all, There probably aren't a lot of people in this situation nowadays, but for those that are, I wanted to report a successful nova-network to Neutron migration. We used NeCTAR's migration scripts which can be found here: https://github.com/NeCTAR-RC/novanet2neutron These scripts allowed us

Re: [Openstack-operators] Successful nova-network to Neutron Migration

2017-05-20 Thread Joe Topjian
n the migration, > like us :) > > thanks, > Belmiro > > On Sat, May 20, 2017 at 6:51 PM, Joe Topjian wrote: > >> Hi all, >> >> There probably aren't a lot of people in this situation nowadays, but for >> those that are, I wanted to report a succe

[Openstack-operators] mitaka/xenial libvirt issues

2017-11-23 Thread Joe Topjian
Hi all, We're seeing some strange libvirt issues in an Ubuntu 16.04 environment. It's running Mitaka, but I don't think this is a problem with OpenStack itself. We're in the process of upgrading this environment from Ubuntu 14.04 with the Mitaka cloud archive to 16.04. Instances are being live mi

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-23 Thread Joe Topjian
MU and related packages at the following version (you might > need to hunt down the debs manually): > > 1:2.5+dfsg-5ubuntu10.5 > > I'm certain there's a launchpad bug for Ubuntu qemu regarding this, but > don't have it to hand. > > Hope this helps, > Chris >

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-23 Thread Joe Topjian
11:03 AM, Chris Sarginson wrote: > I think we may have pinned libvirt-bin as well, (1.3.1), but I can't > guarantee that, sorry - I would suggest its worth trying pinning both > initially. > > Chris > > On Thu, 23 Nov 2017 at 17:42 Joe Topjian wrote: > >> Hi Chris, >

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-27 Thread Joe Topjian
t; https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1647389 > > Thanks > > On Thu, Nov 23, 2017 at 6:20 PM, Joe Topjian wrote: > >> OK, thanks. We'll definitely look at downgrading in a test environment. >> >> To add some further info to this problem, here

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-27 Thread Joe Topjian
We think we've pinned the qemu errors down to a mismatched group ID on a handful of compute nodes. The slow systemd/libvirt is still unsolved, but at the moment that does not actually be the cause of the qemu errors. On Mon, Nov 27, 2017 at 8:04 AM, Joe Topjian wrote: > Hi all, &

Re: [Openstack-operators] [Openstack] Certifying SDKs

2017-12-15 Thread Joe Topjian
Hi all, I've been meaning to reply to this thread. Volodymyr, your reply reminded me :) I agree with what you said that the SDK should support everything that the API supports. In that way, one could simply review the API reference docs and create a checklist for each possible action. I've often

Re: [Openstack-operators] large high-performance ephemeral storage

2018-06-13 Thread Joe Topjian
Yes, you can! The kernel documentation for read/write limits actually uses /dev/null in the examples :) But more seriously: while we have not architected specifically for high performance, for the past few years, we have used a zpool of cheap spindle disks and 1-2 SSD disks for caching. We have ZF

Re: [Openstack-operators] large high-performance ephemeral storage

2018-06-13 Thread Joe Topjian
, and some sort of r/w mix (where write <=50% > to acknowledge this is ephemeral storage so hopefully something is using it > soon after storing). Thoughts? > > Cheers, > Blair > > On Thu., 14 Jun. 2018, 00:24 Joe Topjian, wrote: > >> Yes, you can! The kernel docume

Re: [Openstack-operators] FAiled to create instance wiht openstack nova network

2015-07-13 Thread Joe Topjian
Hello, According to nova.conf, you're running nova-network in multi-host mode. Just to be verbose: if your OpenStack installation is an all-in-one or if you intend for all network traffic to go through the cloud controller, this setting should be changed to false. The error message is reporting:

Re: [Openstack-operators] Palo Alto Midcycle - agenda brainstorming

2015-07-18 Thread Joe Topjian
Hi Tom, The list of General Session ideas is definitely shorter than past meetups, but maybe that's a good sign! It could be that past burning topics have been acknowledged and handled. If that's the case, does anyone have thoughts about extending the length of Working Group sessions so there's m

[Openstack-operators] Compressed Images

2015-07-29 Thread Joe Topjian
Hello, In the "Create An Image" page of Horizon, it says the following: Currently only images available via an HTTP URL are supported. The image location must be accessible to the Image Service. Compressed image binaries are supported (.zip and .tar.gz.) Either I have something misconfigured, th

Re: [Openstack-operators] Compressed Images

2015-07-30 Thread Joe Topjian
I can imagine that '.zip and .tar.gz' might have applied to raw, which can > be quite large, but ultimately, you're probably right and found text that > needs updating. > > > On Jul 29, 2015, at 12:38 PM, Joe Topjian wrote: > > > > Hello, > > > > In

Re: [Openstack-operators] Draft Agenda for PAO Ops Meetup (August 18, 19)

2015-08-13 Thread Joe Topjian
Hi Tom, On Thu, Aug 13, 2015 at 2:08 AM, Tom Fifield wrote: > Hi all, > > We're still lacking moderators for the following sessions - any takers? > > 1. Hypervisor Tuning - General Session (have a backup, but would like a > primary) > I'm happy to be the primary on this -- not a problem at all

Re: [Openstack-operators] [openstack-operators] Announcement! We have everything ready to get to Stackforge!

2015-09-02 Thread Joe Topjian
Hi JJ, Thank you for putting all of this together! All, one decision that was made during the PAO Ops Meetup was to make the OSOps github repos "official" repos / projects. If you've contributed to the existing repos, you may have an interest in this. As well, if you p

Re: [Openstack-operators] Tokyo Summit Ops Design Summit Tracks - Draft Agenda

2015-09-22 Thread Joe Topjian
Hi Tom, As luck would have it, the sole session I'm doing in the Main Conference clashes with the current time slot for Hypervisor Tuning. I see that Tim is also listed as a moderator for that session, so I'm more than happy to let him run it. However, if it's not asking much, I'd really like to b

Re: [Openstack-operators] Tokyo Summit Ops Design Summit Tracks - Draft Agenda

2015-09-23 Thread Joe Topjian
Thanks, guys! On Wed, Sep 23, 2015 at 1:33 AM, Tom Fifield wrote: > Moved it to a day earlier. > > > Regards, > > > Tom > > On 23/09/15 14:25, Tim Bell wrote: > >> I’d like Joe there too JCan we re-schedule that one ? >> >> Tim >> >>

Re: [Openstack-operators] [openstack-operators][osops] Something other than NOOP in our jenkins tests

2015-09-29 Thread Joe Topjian
So this will require bash scripts to adhere to bashate before being accepted? Is it possible to have the check as non-voting? Does this open the door to having other file types be checked? IMHO, it's more important for the OSOps project to foster collaboration and contributions rather than worry a

Re: [Openstack-operators] [openstack-operators][osops] Something other than NOOP in our jenkins tests

2015-09-29 Thread Joe Topjian
people could share what they had. And a curated one, > where everything within the curated repo follows a standard set of > conventions/guidelines. > > _______ > Kris Lindgren > Senior Linux Systems Engineer > Go

[Openstack-operators] Hypervisor Tuning Guide

2015-10-27 Thread Joe Topjian
We had a great Hypervisor Tuning Guide session yesterday! We agreed on an initial structure to the guide that will include four core sections (CPU, Memory, Network, and Disk) and common subsections to each. The etherpad[1] has this structure defined and during the session, we went through and adde

Re: [Openstack-operators] Informal Ops Meetup?

2015-10-29 Thread Joe Topjian
Hi Kris, I'll be around and am interested. On Thu, Oct 29, 2015 at 4:39 PM, Kris G. Lindgren wrote: > Hello all, > > I am not sure if you guys have looked at the schedule for Friday… but its > all working groups. I was talking with a few other operators and the idea > came up around doing an i

Re: [Openstack-operators] Informal Ops Meetup?

2015-10-29 Thread Joe Topjian
We're currently in the Prince room by the projector. On Fri, Oct 30, 2015 at 10:07 AM, Edgar Magana wrote: > Where are you meeting? > > Edgar > > From: "Kris G. Lindgren" > Date: Thursday, October 29, 2015 at 6:37 AM > To: Sam Morrison , " > openstack-operators@lists.openstack.org" < > openstac

Re: [Openstack-operators] OpenStack Tuning Guide

2015-11-04 Thread Joe Topjian
Hi Kevin, Oops, noticed I didn't reply to all the first time. I think it's great to see more people who want to collect and distil knowledge like this. :) Finally, I hate diverging resources, so if something like this already > exists please speak up so we can focus our efforts on making sure th

Re: [Openstack-operators] OPs Midcycle location discussion.

2015-11-16 Thread Joe Topjian
+1 Option 1 On Mon, Nov 16, 2015 at 10:01 AM, Jonathan Proulx wrote: > > Let me restate the question a bit as I think I'm hearing two different > responses that may be getting conflated. > > Option 1: There's a single Ops Midcycle that shifts around and we > look at ways to increase remote part

Re: [Openstack-operators] [Nova] Question about starting nova as service versus directly

2015-11-19 Thread Joe Topjian
Hi Adam, I've seen this happen due to permission issues. Regardless of running with sudo, upstart is dropping to the "nova" user. I usually debug this by setting a shell on the nova user, sudoing/su'ing to nova, then running nova-compute from there. It should die with an error message of the caus

Re: [Openstack-operators] [openstack-operators][osops] tools-contrib is open for business!

2015-11-19 Thread Joe Topjian
Thanks, JJ! It looks like David Wahlstrom submitted a script and there's a question about license. https://review.openstack.org/#/c/247823/ Though contributions to contrib do not have to follow a certain coding style, can be very lax on error handling, etc, should they at least mention a license

Re: [Openstack-operators] [openstack-operators][osops] tools-contrib is open for business!

2015-11-19 Thread Joe Topjian
scher" wrote: > >> Is there a reason why we can't license the entire repo with Apache2 and >> if you want to contribute you agree to that? Otherwise it might become a >> bit of a nightmare. Or maybe at least do "Apache2 unless otherwise stated"? >&g

Re: [Openstack-operators] [Nova] Question about starting nova as service versus directly

2015-11-20 Thread Joe Topjian
> Yes, most likely is related to permissions. Another good source of > information for troubleshooting is /var/log/upstart/nova-compute.log > Ah yes! Much easier. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.o

Re: [Openstack-operators] Router associated with multiple l3 agents

2015-11-24 Thread Joe Topjian
Hi Matt, > It's also weird that we've only seen this when the environment has been > built using terraform. This particular customer re-creates the issue every > time they rebuild. > I work on the OpenStack support for Terraform, so I might be able to help with this. Could you provide an example T

Re: [Openstack-operators] instances floating IPs not reachable while restarting nova-network

2015-11-26 Thread Joe Topjian
Yup, this is expected. It happens for both single-host and multi-host. With the former, we have an older environment where it takes around 10 minutes for all network access to resume. That's with a few hundred tenants, a few hundred vlans, and a few hundred floating IPs all on one host, though. >F

[Openstack-operators] Horizon Kilo bug with nova-network and floating IPs

2015-11-27 Thread Joe Topjian
Hi all, I recently came across this bug and thought I'd share it for anyone else running a similar environment: https://bugs.launchpad.net/horizon/+bug/1520071 Joe ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://list

Re: [Openstack-operators] Hypervisor Tuning Guide

2015-12-08 Thread Joe Topjian
this guide fall under a more formal schedule and would like to lead that effort, please get in contact with me. Thanks, Joe 1: https://wiki.openstack.org/wiki/Documentation/HypervisorTuningGuide 2: https://wiki.openstack.org/wiki/Documentation/HypervisorTuningGuide#How_to_Contribute On Tue, Oct 27,

Re: [Openstack-operators] Cinder API with multiple regions not working.

2015-12-09 Thread Joe Topjian
Hi Salman, Someone mentioned this same issue yesterday in relation to Terraform (maybe a colleague of yours?), so given the two occurrences, I thought I'd look into this. I have a Liberty environment readily available, so I created a second set of volume and volumev2 endpoints for a fictional reg

Re: [Openstack-operators] Cinder API with multiple regions not working.

2015-12-10 Thread Joe Topjian
| > > > +--+--+ > —— > > Again if I remove hpc2n everything started to work fine. Can you tell me > what service I suppose to restart (if required)? > > Regards.. > Salman > > > > PhD, Sc

Re: [Openstack-operators] Galera setup testing

2015-12-11 Thread Joe Topjian
We do something similar: Instead of McRouter, we use the repcached patches to replicate data between two memcached nodes. We then use HAProxy as a single entry point for memcached requests. We've been doing this for 6+ months and it's been working great. It's effectively solved the issue I describ

Re: [Openstack-operators] Cinder API with multiple regions not working.

2015-12-11 Thread Joe Topjian
t; —— cinder.conf > [default] > os_region_name = regionOne > —— > > Regards.. > Salman. > > > > > > PhD, Scientific Computing > Researcher, IT Department, > Uppsala University. > Senior Cloud Architect, > SNIC. > Cloud Application Expert, > U

Re: [Openstack-operators] Cinder API with multiple regions not working.

2015-12-11 Thread Joe Topjian
61686 | regionOne | glance | image > | > > +--+---+--+--+ > > > > But different names are working perfectly fine with glance and nova. > > Anyways thanks for all your time and efforts. If I manage to rep

Re: [Openstack-operators] Storage backend for glance

2016-01-27 Thread Joe Topjian
Yup, it's definitely possible. All Glance nodes will need to share the same database as well as the same file system. Common ways of sharing the file system are to mount /var/lib/glance/images either from NFS (like you mentioned) or Gluster. I've done both in the past with no issues. The usual cav

[Openstack-operators] RAID / stripe block storage volumes

2016-02-05 Thread Joe Topjian
Hello, Does anyone have users RAID'ing or striping multiple block storage volumes from within an instance? If so, what was the experience? Good, bad, possible but with caveats? Thanks, Joe ___ OpenStack-operators mailing list OpenStack-operators@lists.

Re: [Openstack-operators] RAID / stripe block storage volumes

2016-02-08 Thread Joe Topjian
> I've always recommended providing multiple underlying storage services to > provide this rather than adding the overhead to the VM. So, not in any of > my systems or any I've worked with. > > R > > > > On Fri, Feb 5, 2016 at 5:56 PM, Joe Topjian wrote: > >>

Re: [Openstack-operators] RAID / stripe block storage volumes

2016-02-08 Thread Joe Topjian
te environments. >>> >>> We've been discussing how possible it would be to translate that same >>> idea to block storage. Rather than have a large HA storage cluster (whether >>> Ceph, Gluster, NetApp, etc), is it possible to offer simple single LVM >&g

Re: [Openstack-operators] RAID / stripe block storage volumes

2016-02-08 Thread Joe Topjian
gt;> I have not run into anyone replicating volumes or creating redundancy at >> the VM level (beyond, as you point out, HDFS, etc.). >> >> R >> >> On Mon, Feb 8, 2016 at 6:54 PM, Joe Topjian wrote: >> >>> This is a great conversation and I really a

Re: [Openstack-operators] User Survey - Deadline Feb 24th

2016-02-09 Thread Joe Topjian
Isn't it similar to asking if you use Fedora, CentOS, or RHEL? My understanding is that Juniper offers a paid/supported version of Contrail while OpenContrail is the open source version. On Tue, Feb 9, 2016 at 2:50 AM, Edgar Magana wrote: > Tom, > > For the "Which OpenStack Network (Neutron) dr

Re: [Openstack-operators] User Survey - Deadline Feb 24th

2016-02-09 Thread Joe Topjian
ot Enterprise version of OpenContrail. > Therefore, this question could split the responses because it is confusing. > > Edgar > > From: Joe Topjian > Date: Tuesday, February 9, 2016 at 8:17 AM > To: Edgar Magana > Cc: Tom Fifield , "openst...@lists.openstack.org"

Re: [Openstack-operators] [kolla] Question about how Operators deploy

2016-02-12 Thread Joe Topjian
2 VIPs as well. On Fri, Feb 12, 2016 at 8:27 AM, Matt Fischer wrote: > We also use 2 VIPs. public and internal, with admin being a CNAME for > internal. > > On Fri, Feb 12, 2016 at 7:28 AM, Fox, Kevin M wrote: > >> We usually use two vips. >> >> Thanks, >> Kevin >> >> --

Re: [Openstack-operators] RAID / stripe block storage volumes

2016-03-06 Thread Joe Topjian
rticles/virtual-machines-linux-configure-raid/ 4: https://wiki.openstack.org/wiki/LibvirtVirtioScsi On Mon, Feb 8, 2016 at 7:18 PM, Joe Topjian wrote: > Yep. Don't get me wrong -- I agree 100% with everything you've said > throughout this thread. Applications that have native replication a

Re: [Openstack-operators] RAID / stripe block storage volumes

2016-03-07 Thread Joe Topjian
re-raid-on-a-linux-general-purpose-cloud-server/ > 2: https://support.rackspace.com/how-to/cloud-block-storage-faq/ > 3: > https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-configure-raid/ > 4: https://wiki.openstack.org/wiki/LibvirtVirtioScsi > >

Re: [Openstack-operators] How are folks providing GPU instance types?

2016-05-11 Thread Joe Topjian
Just wanted to add a few notes (I apologize for the brevity): * The wiki page is indeed the best source of information to get started. * I found that I didn't have to use EFI-based images. I wonder why that is? * PCI devices and IDs can be found by running the following on a compute node: $ lspci

Re: [Openstack-operators] PCI Passthrough issues

2016-07-06 Thread Joe Topjian
Hi Jon, We were also running into issues with the K80s. For our GPU nodes, we've gone with a 4.2 or 4.4 kernel. PCI Passthrough works much better in those releases. (I ran into odd issues with 4.4 and NFS, downgraded to 4.2 after a few hours of banging my head, problems went away, not a scientifi

Re: [Openstack-operators] How to create floating ip pool use nova network? thanks

2016-07-07 Thread Joe Topjian
In Kilo (I haven't verified Liberty or Mitaka), you can manage nova-network floating IP pools with: nova-manage floating --help nova-manage floating create --help Hope that helps! Joe On Wed, Jul 6, 2016 at 8:23 PM, 云淡风轻 <821696...@qq.com> wrote: > hi everyone, > > How to create floating ip po

Re: [Openstack-operators] PCI Passthrough issues

2016-07-19 Thread Joe Topjian
Hi Blair, We only updated qemu. We're running the version of libvirt from the Kilo cloudarchive. We've been in production with our K80s for around two weeks now and have had several users report success. Thanks, Joe On Tue, Jul 19, 2016 at 5:06 PM, Blair Bethwaite wrote: > Hilariously (or not

Re: [Openstack-operators] cURL call at the end of provisioning and deprovisioning

2016-08-15 Thread Joe Topjian
Hi Will, What about notification events? I'm not sure what the best source of documentation is for events, but googling "OpenStack Notification Events" yields a bunch of information that should lead you in the right direction. With events, you would write a custom scripts / daemon that polls rabb

[Openstack-operators] Murano in Production

2016-09-16 Thread Joe Topjian
Hi all, We're planning to deploy Murano to one of our OpenStack clouds and I'm debating the RabbitMQ setup. For background: the Murano agent that runs on instances requires access to RabbitMQ. Murano is able to be configured with two RabbitMQ services: one for traditional OpenStack communication

Re: [Openstack-operators] Murano in Production

2016-09-18 Thread Joe Topjian
burns on us, I'll be more than happy to report failure. :) On Sun, Sep 18, 2016 at 7:38 PM, Silence Dogood wrote: > I'd love to see your results on this . Very interesting stuff. > > On Sep 17, 2016 1:37 AM, "Joe Topjian" wrote: > >> Hi all, >> >

Re: [Openstack-operators] Murano in Production

2016-09-18 Thread Joe Topjian
m Morrison wrote: > You could also use https://www.rabbitmq.com/maxlength.html to mitigate > overflowing on the trove vhost side. > > > Sam > > > On 19 Sep 2016, at 1:07 PM, Joe Topjian wrote: > > Thanks for everyone's input. I think I'm going to go wit

Re: [Openstack-operators] Murano in Production

2016-09-23 Thread Joe Topjian
Hi Serg, Thank you for sharing this information :) If I'm understanding correctly, the main reason you're using a non-clustered / corosync setup is because that's how most other components in Mirantis OpenStack are configured? Is there anything to be aware of in how Murano communicates over the a

Re: [Openstack-operators] Murano in Production

2016-09-26 Thread Joe Topjian
Hi Serg, We were indeed hitting that bug, but the cert wasn't self-signed. It was easier for us to manually patch the Ubuntu Cloud package of Murano with the stable/mitaka fix linked in that bug report than trying to debug where OpenSSL/python/requests/etc was going awry. We might redeploy Murano

Re: [Openstack-operators] [nova] Does anyone use the os-diagnostics API?

2016-10-12 Thread Joe Topjian
Hi Matt, Tim, Thanks for asking. We’ve used the API in the past as a way of getting the > usage data out of Nova. We had problems running ceilometer at scale and > this was a way of retrieving the data for our accounting reports. We > created a special policy configuration to allow authorised user

Re: [Openstack-operators] RabbitMQ issues since upgrading to Icehouse

2014-09-02 Thread Joe Topjian
Hi Sam, We upgraded to Icehouse over the weekend and had some issues with Rabbit. The number of RabbitMQ connections went from ~70 to ~380 post-upgrade and we had reports of users being unable to use the metadata service and that instances took longer to boot as they waited for metadata. I notic

Re: [Openstack-operators] Openstack Manual install with nova-network

2014-09-03 Thread Joe Topjian
Hello, Do you have an entry in the Keystone Catalog for Neutron? If so, remove it. I believe Horizon will attempt to enable services based on what the catalog contains. For example, if you have a Swift entry in the catalog, Horizon will display the Swift menu items. Thanks, Joe On Wed, Sep 3, 2

Re: [Openstack-operators] Fwd: Unexpected error in OpenStack Nova

2014-09-04 Thread Joe Topjian
Hello, You have your OS_TENANT_NAME set to "service" for the admin user. The installation docs don't place the admin user in the service tenant. Can you try changing OS_TENANT_NAME to "admin" (assuming you placed the admin user into the admin tenant) and see if that helps? Please see here for mor

Re: [Openstack-operators] Openstack and mysql galera with haproxy

2014-09-18 Thread Joe Topjian
Hello, I'm using MySQL (Percona) and Galera with HAProxy and have previously run into the same issues you describe. The first thing to mention is "SELECT ... FOR UPDATE" queries do not work well with multi-master environments. This was discovered during discussion at the OpenStack Atlanta Summit.

Re: [Openstack-operators] Nodes and configurations management in Puppet

2014-09-25 Thread Joe Topjian
Hi Mathieu, My setup is very similar to yours. Node definitions are in site.pp and Hiera is used for all configuration. The Hiera hierarchies are also very similar. Overall, I have a love/hate relationship with the setup. I could go on in detail, but it'd be all Puppet-specific rather than OpenSt

Re: [Openstack-operators] Nodes and configurations management in Puppet

2014-09-25 Thread Joe Topjian
Stack components run in LXC containers. Our Icehouse upgrade procedure consisted of killing those containers, making new ones based on Ubuntu 14.04, and letting Puppet do the rest. Let me know if you'd like more details or clarification. Thanks, Joe On Thu, Sep 25, 2014 at 12:12 PM,

Re: [Openstack-operators] Problem creating resizable CentOS 6.5 image

2014-10-06 Thread Joe Topjian
Does this cover the scenario of a user launching CentOS 6.x, updating the kernel, snapshotting, and having the relaunched instance resized? On Mon, Oct 6, 2014 at 12:38 PM, Regan McDonald wrote: > Seconded. This is what I use with my CentOS images, and it works great. > > > On Mon, Oct 6, 2014

Re: [Openstack-operators] [glance] how to update the contents of an image

2014-10-08 Thread Joe Topjian
We just ran some tests on our production Icehouse environment and can confirm that: * Snapshotting an instance based on a deleted image works * Snapshotting an instance based on a public-turned-private image works * Block migration of an instance based on a deleted image works This environment do

Re: [Openstack-operators] [glance] how to update the contents of an image

2014-10-08 Thread Joe Topjian
e raw disk of instance booted from snapshot it > gonna be double migration time). > On Oct 8, 2014 8:19 PM, "Joe Topjian" wrote: > >> We just ran some tests on our production Icehouse environment and can >> confirm that: >> >> * Snapshotting an instance

[Openstack-operators] rsyslog update caused services to break?

2014-10-10 Thread Joe Topjian
Hello, This morning we noticed various nova, glance, and keystone services (not cinder or swift) not working in two different clouds and required a restart. We thought it was a network issue since one of the only commonalities between the two clouds was that they are on the same network. Then la

[Openstack-operators] rsyslog update caused services to break?

2014-10-10 Thread Joe Topjian
Hello, This morning we noticed various nova, glance, and keystone services (not cinder or swift) not working in two different clouds and required a restart. We thought it was a network issue since one of the only commonalities between the two clouds was that they are on the same network. Then la

Re: [Openstack-operators] rsyslog update caused services to break?

2014-10-13 Thread Joe Topjian
it thanks to the following script. > > https://gist.github.com/dbishop/7a2e224f3aafea1a1fc3 > > François > > On 11. 10. 14 00:40, Joe Topjian wrote: > > Hello, > > > > This morning we noticed various nova, glance, and keystone services (not > > cinder or swift) not

[Openstack-operators] state of gpu processing / passthrough

2014-10-16 Thread Joe Topjian
Hi all, I did a standard google search and got a bunch of wiki pages, but I'm not sure if they are stale. Does anyone know what the state of GPU processing is in OpenStack? I could have swore I read it was available for Xen, but what about KVM? Is anyone doing it? If so, what hardware are you us

Re: [Openstack-operators] state of gpu processing / passthrough

2014-10-16 Thread Joe Topjian
PU > passthrough with NVIDIA cards and libvirt/KVM. This may have changed though. > > BR, > Simon > > [1] https://blueprints.launchpad.net/nova/+spec/pci-passthrough-xenapi > [2] > http://blogs.citrix.com/2014/03/06/hpc-cloud-enablement-using-xenserver-openstack-and-nvidia-g

[Openstack-operators] qemu 1.x to 2.0

2014-10-19 Thread Joe Topjian
Hello, We recently upgraded an OpenStack Grizzly environment to Icehouse (doing a quick stop-over at Havana). This environment is still running Ubuntu 12.04. The Ubuntu 14.04 release notes make mention of incompatibilities with 12.04

Re: [Openstack-operators] qemu 1.x to 2.0

2014-10-21 Thread Joe Topjian
> I wonder if the qemu update issues are in the realm of the OpenStack > project. > Since they are provided / packaged by the distro's. > Maybe people from Ubuntu & Redhat (which I would be interested in ;) > reading this could give some suggestions... > I'd really appreciate someone from Ubuntu o

Re: [Openstack-operators] Console security question when using nova-novncproxy to access console

2014-10-22 Thread Joe Topjian
Hi Niall, It looks like vnc password support was removed from the vmware driver last October: https://github.com/openstack/nova/commit/058ea40e7b7fb2181a2058e6118dce3f051e1ff3 For libvirt, there is an option in qemu.conf for "vnc_password", but I'm not sure how it would work with OpenStack. Tha

Re: [Openstack-operators] Three Node Deployment Model

2014-10-23 Thread Joe Topjian
Hi Hossein, This is a pretty open-ended question that can be answered several different ways. Have you read through both the Operators Guide and Architecture Design Guide ? Both of those docs might

[Openstack-operators] Guaranteed Resources

2014-10-23 Thread Joe Topjian
Hello, I'm sure some of you have run into this situation before and I'm wondering how you've dealt with it: A user requests that they must have access to a certain amount of resources at all times. This is to prevent them from being unable to launch instances in the cloud when the cloud is at ful

Re: [Openstack-operators] [nova] instance resource quota quesetions

2014-10-23 Thread Joe Topjian
I can confidently say that throttling will work with KVM. I think both virt_types will work since libvirt is controlling everything in the end. One caveat about IO throttling to keep in mind is that the Nova settings are not applied to volumes -- just the root and ephemeral disk. We were unable to

Re: [Openstack-operators] Guaranteed Resources

2014-10-24 Thread Joe Topjian
ill > unshelve the instances. > HTH > Simon > [1] https://wiki.openstack.org/wiki/Blazar#Virtual_instance_reservation > > On Thu, Oct 23, 2014 at 5:20 PM, Joe Topjian wrote: > >> Hello, >> >> I'm sure some of you have run into this situation before and I&

Re: [Openstack-operators] qemu 1.x to 2.0

2014-10-29 Thread Joe Topjian
Hi all, I had some more fun with this this morning. libvirt 1.2.2-0ubuntu13.1.6 became available in the Ubuntu cloud archive precise-updates repository. Installing it on compute nodes removed apparmor. When trying to restart an instance, libvirt complained that it couldn't find the apparmor profi

Re: [Openstack-operators] OpenStack Storage Backend: Sheepdog, Ceph, or GlusterFS

2014-11-07 Thread Joe Topjian
Gluster indeed provides both block and object storage. We use the Gluster Cinder driver in production and, short of some initial hiccups, it's been running great. I have no experience with Gluster's object storage suppor

Re: [Openstack-operators] OpenStack Storage Backend: Sheepdog, Ceph, or GlusterFS

2014-11-10 Thread Joe Topjian
> About GlusterFS, I found a blog talked about Gluster limitation and > want to share it with others: > > https://shellycloud.com/blog/2013/09/why-glusterfs-should-not-be-implemented-with-openstack It's unfortunate that that blog post appears in so many gluster+openstack google searches. As per

Re: [Openstack-operators] Proposal for an 'Operations' project

2014-11-10 Thread Joe Topjian
> > I fear that people contributing scripts to osops/* will forget about them > and scripts will go stalled over time. > That's a fair assessment and that's probably what will happen to a lot of the items added to those repositories. But really any open source project is susceptible to that. Both

Re: [Openstack-operators] Openstack telemetry stuff

2014-11-10 Thread Joe Topjian
Hi Kris, We're not collecting metrics on api requests, response times, etc, but we do collect a fair amount of stats about the hypervisors and cloud usage. We use Sensu as our metric collector and it works pretty well. It feeds the metrics into Rabbit where graphite then picks them up. We have an

Re: [Openstack-operators] qemu 1.x to 2.0

2014-11-17 Thread Joe Topjian
hosts), but it looks as though the latest Ubuntu packages resolve any backwards compatibility issues. On Mon, Nov 17, 2014 at 9:12 PM, Felipe Reyes wrote: > Hi, > > On Tue, 21 Oct 2014 13:42:24 -0600 > Joe Topjian wrote: > > > > I wonder if the qemu update issues are

Re: [Openstack-operators] qemu 1.x to 2.0

2014-11-18 Thread Joe Topjian
#x27;m seeing these in APT: > 1.0.50-2012.03-0ubuntu2 > 1.0.50-2012.03-0ubuntu2.1 > 1.5.0+dfsg-3ubuntu5.4~cloud0 (havana ubuntu cloud repo) > 2.0.0+dfsg-2ubuntu1.6~cloud0 (icehouse ubuntu cloud repo) > > Thanks, > Daniele > > > On 11/17/14 21:30, Joe Topjian wrote: >

Re: [Openstack-operators] qemu 1.x to 2.0

2014-11-18 Thread Joe Topjian
~cloud2 libvirt0 1.2.2-0ubuntu13.1.6~cloud2 python-libvirt 1.2.2-0ubuntu2~cloud0 Joe On Tue, Nov 18, 2014 at 10:49 AM, Joe Topjian wrote: > Hi Daniele, > > Here are the packages I started with in my test environment and what the > initial instances were based on: > > dpkg -l | g

Re: [Openstack-operators] qemu 1.x to 2.0

2014-11-26 Thread Joe Topjian
" so it's now compatible with 14.04. So as far as I can tell, I'm in good shape. I've tested a bunch of scenarios (launching, migrating, rebooting, etc) and everything seems OK. I'll give it a few days to see if any users run into issues. Thanks, Joe On Tue, Nov 18, 2

  1   2   >