This is a bit delayed due to the release rush, finally getting back to writing up my experiences at the Ops Meetup.
Nova Feedback Session ===================== We had a double session for Feedback for Nova from Operators, raw etherpad here - https://etherpad.openstack.org/p/NYC-ops-Nova. The median release people were on in the room was Kilo. Some were upgrading to Liberty, many had older than Kilo clouds. Remembering these are the larger ops environments that are engaged enough with the community to send people to the Ops Meetup. Performance Bottlenecks ----------------------- * scheduling issues with Ironic - (this is a bug we got through during the week after the session) * live snapshots actually end up performance issue for people The workarounds config group was not well known, and everyone in the room wished we advertised that a bit more. The solution for snapshot performance is in there. There were also general questions about what scale cells should be considered at. ACTION: we should make sure workarounds are advertised better ACTION: we should have some document about "when cells"? Networking ---------- A number of folks in the room were still on Nova Net, and were a bit nervous about it going away. As they are Kilo / Liberty it's still a few upgrades before they get there, but that nervousness and concern was definitely there. Policy ------ How are you customizing policy? People were largely making policy changes to protect their users that didn't really understand cloud semantics. Turning off features that they thought would confuse them (like pause). The large number of VM states is confusing, and not clearly useful for end users, and they would like simplification. Ideally policy could be set on a project by project admin, because they would like to delegate that responsibility down. No one was using the user_id based custom policy (yay!). There was desire that flavors could be RBAC locked down, which was actually being done via policy hacks right now. Providers want to expose some flavors (especially those with aggregate affinity) to only some projects. People were excited about the policy in code effort, only concern was that the defacto documentation of what you could change wouldn't be in the sample config. ACTION: ensure there is policy config reference now that the sample file is empty ACTION: flavor RBAC is a thing most of the room wanted, is there a taker on spec / implementation? Upgrade ------- Everyone waits to do any optional thing until they absolutely have to. The Cells API db caught a bunch of people off guard because it was in Kilo (with release note) optional. Status quo in Liberty, with no release note about it existing, then forced in Mitaka. When an optional component is out there make sure it continues to be talked about in releases even when it's status did not change, or people forget. People were on Kilo, so EC2 out of tree didn't really have any data. About 25% of folks users have some existing AWS tooling, that it's good to be able to just let them use to onboard them. The current DB online data upgrade model feels *very opaque* to ops. They didn't realize the current model Nova was using, and didn't feel like it was documented anywhere. ACTION: document the DB data lifecycle better for operators ACTION: make sure we are cautious in rewarning people about changes they have to make (like Cells API db) API --- API upgrade seemed fine for folks. The only question was the new policy names, which was taking folks a bit of time to adjust to. No one in the room was using custom API extensions (or at least admitted to it when I asked). Tracking Feedback ----------------- We talked a bit about tracking feedback. The silence on the ops list mostly comes from people not using a particular feature, so they don't really have an opinion. Most ops do not have time to look at our specs. That is an unlikely place to get feedback. Additional Questions -------------------- There was an ask about VM HA. I stated that was beyond scope for Nova, plus Nova's view of the world is non authoritative enough you didn't want it to do that anyway. I told folks that the NFV efforts were working on this kind of thing beyond Nova, and people should team up there. There was an ask on status of Cinder Multi Attach. We gave them a bit of status on where things were at. ACTION: Cinder Multi Attach should maybe be a priority effort in the next cycle. Upgrade Pain Points =================== Raw etherpad - https://etherpad.openstack.org/p/NYC-ops-Upgrades-Pain-points Most people are a couple of releases back (Kilo / Liberty or even older). The only team CDing in the room was RAX, they are now 2 to 3 months behind master. Everyone agrees upgrades are getting better with every release. Most are taking change windows and downtime for upgrades. Why are upgrades taking so long? -------------------------------- About half way through this session I threw this powder keg into the room, it generated a lot of feedback. People are holding a lot of out of tree patches, causing latency. These are for: * bug fixes that get fixed on old versions of OpenStack, so upstream won't take them (chicken / egg problem of being on an old release) * custom identity driver for keystone * some new feature that customer wants, not taken upstream * lack of time to invest working patches upstream (Thinking out loud, I do wonder if there is a way we could close this gap) Defcore is actually forcing upgrades, because people loose their OpenStack trademark if they don't stay in the Defcore supported window. Other Sessions ============== There were a ton of other sessions there. Interesting things that I remember from them. In the session on deploying OpenStack in containers, there was a split between idempotent (docker) vs. system (lxc/lxd) containers. Both were getting used in different ways, and there was debate between the camps as to which was most effective. Ceilometer deployments are highly coupled to Heat, and seem to be only used when users want Heat auto scaling. There are noticable failures in our CLI / API paths when using UTF8 names for projects / resources. Noticed by many .eu folks. Would be good to increase testing in areas like this. The full list of all etherpads are here for anyone else looking to dive in and learn more - https://etherpad.openstack.org/p/NYC-ops-meetup -Sean -- Sean Dague http://dague.net __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev