Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Filip, If I got you right the plan is to have Murano application execute Mistral workflow that SSH to VM and executes particular command? And alternative is Murano-Mistral-Zaquar-Zaquar agent? Why can't you just send this command directly from Murano (to Murano agent on VM)? This is the most common use case that is found in nearly all Murano applications and it is battle-proven. If you need SSH you can contribute SSH plugin to Murano (Mistral will require similar plugin anyway). The more moving parts you involve the more chances you have for everything to fail Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Fri, May 8, 2015 at 11:22 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] global-requirements not co-installable
On 2015-05-10 07:49:25 +1200 (+1200), Robert Collins wrote: [...] We should do the following immediately we create a virtualenv anywhere in our infra: get-pip.py pip install -U wheel setuptools [...] So... we already do install latest pip and setuptools in the system context on our systems. We're switching to the current version of virtualenv real soon now so that anything using tox gets a more recent bundled pip version by default in their envs. But regardless, if you want a particular repo's tox-based jobs to use the most recent version of pip (regardless of the vintage of virtualenv we're providing at any point in time), that repo's tox.ini needs to expressly upgrade pip before installing other things. Unless we switch to always installing whatever the latest virtualenv release is, there's not much other way to accomplish it. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] global-requirements not co-installable
On 10 May 2015 at 08:03, Jeremy Stanley fu...@yuggoth.org wrote: On 2015-05-10 07:49:25 +1200 (+1200), Robert Collins wrote: [...] We should do the following immediately we create a virtualenv anywhere in our infra: get-pip.py pip install -U wheel setuptools [...] So... we already do install latest pip and setuptools in the system context on our systems. We're switching to the current version of virtualenv real soon now so that anything using tox gets a more recent bundled pip version by default in their envs. But regardless, if you want a particular repo's tox-based jobs to use the most recent version of pip (regardless of the vintage of virtualenv we're providing at any point in time), that repo's tox.ini needs to expressly upgrade pip before installing other things. Unless we switch to always installing whatever the latest virtualenv release is, there's not much other way to accomplish it. Donald Stufft says that virtualenv releases 1:1 with pip releases, so we can skip the pip step once we have latest virtualenv installed. We should still add wheel, and we need to add setuptools as well since that, while bundled, doesn't trigger virtualenv releases (yet). If the virtualenv creation is owned by e.g. tox, then we need to put this in all our tox.ini's, which is going to a drag - perhaps we should have it be in a helper (pbr or os-testr perhaps) and call into them from our tox.inis, so we can centrally manage this. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova-docker] Status update
On 1 May 2015 at 16:14, Davanum Srinivas dava...@gmail.com wrote: Anyone still interested in this work? :) * there's a stable/kilo branch now (see http://git.openstack.org/cgit/stackforge/nova-docker/). * CI jobs are running fine against both nova trunk and nova's stable/kilo branch. * there's an updated nova-spec to get code back into nova tree (see https://review.openstack.org/#/c/128753/) To proxy the discussion from the etherpad onto the ML, we need to work out why this lives in nova, given Magnum is the place to do container specific things. LXC in nova means you can do Magnum containers inside Nova containers, and some folks like the idea of that. You can also use containers just like VMs, for where that makes sense. Now whats the reason for adding the Docker driver, given Nova is considering container specific APIs out of scope, and expecting Magnum to own that kind of thing. Thanks, John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][python-heatclient] Does python-heatclient works with keystone sessions?
- Original Message - From: Jay Reslock jresl...@gmail.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Saturday, May 9, 2015 6:42:48 AM Subject: Re: [openstack-dev] [heat][python-heatclient] Does python-heatclient works with keystone sessions? Interestingit is definitely a service endpoint mismatch. UI: http://10.25.17.63:8004/v1/dac1095f448d476e9990046331415cf6 keystoneclient.services.list(): http://10.25.17.63:35357/v3/services/e0a18f2f4b574c75ba56823964a7d7eb What can I do to make these match up correctly? They're network URLs - i can't see anything there. Try using: `openstack catalog list`. Otherwise i'd turn on python's debug logging, something like: import logging logging.basicConfig(level=logging.DEBUG) which will give you a bunch of output - though the service catalog will be hidden because it's part of the token exchange. On Fri, May 8, 2015 at 4:22 PM Jay Reslock jresl...@gmail.com wrote: Hi Jamie, How do I see the service catalog that I am getting back? On Fri, May 8, 2015 at 3:25 AM Jamie Lennox jamielen...@redhat.com wrote: - Original Message - From: Jay Reslock jresl...@gmail.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Friday, May 8, 2015 7:42:50 AM Subject: Re: [openstack-dev] [heat][python-heatclient] Does python-heatclient works with keystone sessions? Thanks very much to both of you for your help! I was able to get to another error now about EndpointNotFound. I will troubleshoot more and review the bugs mentioned by Sergey. -Jason It's nice to see people using sessions for this sort of script. Just as a pointer EndpointNotFound generally means that it couldn't find a url for the service you wanted in the service catalog. Have a look at the catalog you're getting and make sure the heat entry matches what it should, you may have to change the service_type or interface to match. On Thu, May 7, 2015 at 5:34 PM Sergey Kraynev skray...@mirantis.com wrote: Hi Jay. AFAIK, it works, but we can have some minor issues. There several atches on review to improve it: https://review.openstack.org/#/q/status:open+project:openstack/python-heatclient+branch:master+topic:improve-sessionclient,n,z Also as I remember we really had bug mentioned by you, but fix was merged. Please look: https://review.openstack.org/#/c/160431/1 https://bugs.launchpad.net/python-heatclient/+bug/1427310 Which version of client do you use? Try to use code from master, it should works. Also one note: the best place for such questions is openst...@lists.openstack.org or http://ask.openstack.org/ . And of course channel #heat in IRC. Regards, Sergey. On 7 May 2015 at 23:43, Jay Reslock jresl...@gmail.com wrote: Hi, This is my first mail to the group. I hope I set the subject correctly and that this hasn't been asked already. I searched archives and did not see this question asked or answered previously. I am working on a client thing that uses the python-keystoneclient and python-heatclient api bindings to set up an authenticated session and then use that session to talk to the heat service. This doesn't work for heat but does work for other services such as nova and sahara. Is this because sessions aren't supported in the heatclient api yet? sample code: https://gist.github.com/jreslock/a525abdcce53ca0492a7 I'm using fabric to define tasks so I can call them via another tool. When I run the task I get: TypeError: Client() takes at least 1 argument (0 given) The documentation does not say anything about being able to pass session to the heatclient but the others seem to work. I just want to know if this is intended/expected behavior or not. -Jason __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions)
Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
On 30 April 2015 at 18:54, Mike Bayer mba...@redhat.com wrote: On 4/30/15 11:16 AM, Dan Smith wrote: There is an open discussion to replace mysql-python with PyMySQL, but PyMySQL has worse performance: https://wiki.openstack.org/wiki/PyMySQL_evaluation My major concern with not moving to something different (i.e. not based on the C library) is the threading problem. Especially as we move in the direction of cellsv2 in nova, not blocking the process while waiting for a reply from mysql is going to be critical. Further, I think that we're likely to get back a lot of performance from a supports-eventlet database connection because of the parallelism that conductor currently can only provide in exchange for the footprint of forking into lots of workers. If we're going to move, shouldn't we be looking at something that supports our threading model? yes, but at the same time, we should change our threading model at the level of where APIs are accessed to refer to a database, at the very least using a threadpool behind eventlet. CRUD-oriented database access is faster using traditional threads, even in Python, than using an eventlet-like system or using explicit async. The tests at http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ show this.With traditional threads, we can stay on the C-based MySQL APIs and take full advantage of their speed. Sorry to go back in time, I wanted to go back to an important point. It seems we have three possible approaches: * C lib and eventlet, blocks whole process * pure python lib, and eventlet, eventlet does its thing * go for a C lib and dispatch calls via thread pool We have a few problems: * performance sucks, we have to fork lots of nova-conductors and api nodes * need to support python2.7 and 3.4, but its not currently possible with the lib we use? * want to pick a lib that we can fix when there are issues, and work to improve It sounds like: * currently do the first one, it sucks, forking nova-conductor helps * seems we are thinking the second one might work, we sure get py3.4 + py2.7 support * the last will mean more work, but its likely to be more performant * worried we are picking a unsupported lib with little future I am leaning towards us moving to making DB calls with a thread pool and some fast C based library, so we get the 'best' performance. Is that a crazy thing to be thinking? What am I missing here? Thanks, John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Is Live-migration not supported in CONF.libvirt.images_type=lvm case?
I would really to start collecting more of this information. Maybe my expanding this: https://github.com/openstack/nova/blob/master/doc/source/support-matrix.ini http://docs.openstack.org/developer/nova/support-matrix.html Although it feels like it will need an extra dimention adding, possibly a separate table for volume and local storage options that affect live-migrate vs migrate/resize. Ideas on how best to improve this are very welcome, and probably part of the discussion here: http://libertydesignsummit.sched.org/event/3eb290350853e7238697591434075992 Thanks, John On 8 May 2015 at 14:14, Coffman, Joel M. joel.coff...@jhuapl.edu wrote: I think you’re correct: it looks like the change you identified covers only the migrate code path but doesn’t address live migration. As identified in the comments on the bug report [1], it would be beneficial at least to raise to reasonable error message. I also found an abandoned change for the bug: https://review.openstack.org/#/c/80029/ Joel From: Rui Chen [mailto:chenrui.m...@gmail.com] Sent: Friday, May 08, 2015 4:20 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [nova] Is Live-migration not supported in CONF.libvirt.images_type=lvm case? Hi all: I find the bug [1] block/live migration doesn't work with LVM as libvirt storage is marked as 'Fix released', but I don't think this issue really is solved, I check the live-migration code and don't find any logic for handling LVM disk. Please correct me if I'm wrong. In the bug [1] comments, the only related merged patch is https://review.openstack.org/#/c/73387/ , it cover the 'resize/migrate' code path, not live-migration, and I don't think this bug [1] is duplicate with bug [2], they are the different use case, live-migration and migration. So should we reopen this bug and add some documentation to describe that live-migration is not supported in current code? [1]: https://bugs.launchpad.net/nova/+bug/1282643 [2]: https://bugs.launchpad.net/nova/+bug/1270305 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] global-requirements not co-installable
On 2015-05-09 06:46:59 +1200 (+1200), Robert Collins wrote: As I read it, we've got some tooling that isn't PEP-440 compatible (https://www.python.org/dev/peps/pep-0440/#compatible-release defines ~=) and as such we had to rollback the intended use of that. As long as we identify and fix those tools, we should be fine. Did anyone involved with that situation create a bug we can use to track this? I don't think it has anything to do with the choice of cap-or-not. We had (have) virtualenv versions in use in automation which bundle a pip and setuptools too old to support this... lingering fallout from the scramble around pinning these bits of the toolchain at the end of last year while we tried to keep everything running. Clark has proposed https://review.openstack.org/181558 which we can try and then quickly revert if too much explodes. However it's worth noting, as soon as we start adding ~= into requirements files we're basically saying we no longer support installing any of those packages with pip6 or setuptools8 (or with virtualenv12 without an implicit upgrade step in the resulting env). The supporting versions of these tools/libs aren't even 6 months old yet. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Ironic] Large number of ironic driver bugs in nova
On 6 May 2015 at 19:04, John Villalovos openstack@sodarock.com wrote: JohnG, I work on Ironic and would be willing to be a cross project liaison for Nova and Ironic. I would just need a little info on what to do from the Nova side. Meetings to attend, web pages to monitor, etc... I assume I would start with this page: https://bugs.launchpad.net/nova/+bugs?field.tag=ironic And try to work with the Ironic and Nova teams on getting bugs resolved. I would appreciate any other info and suggestions to help improve the process. Thank you for stepping up to help us here. I have added your name on this list: https://wiki.openstack.org/wiki/Nova#People (if you can please add your IRC handle for me, that would be awesome) In terms of whats required, thats really up to what works for you. The top things that come to mind: * Raise ironic questions to nova in nova-meetings (and maybe v.v.) * For ironic features that need exposing in Nova, track those * Help triage Nova's ironic bugs into Nova bugs and Ironic bugs * Try to find folks to fix important ironic bugs But fundamentally, lets just see what works, and evolve the role to match what works for you. I hope that helps. Thanks, John John On Wed, May 6, 2015 at 2:55 AM, John Garbutt j...@johngarbutt.com wrote: On 6 May 2015 at 09:39, Lucas Alvares Gomes lucasago...@gmail.com wrote: Hi I noticed last night that there are 23 bugs currently filed in nova tagged as ironic related. Whilst some of those are scheduler issues, a lot of them seem like things in the ironic driver itself. Does the ironic team have someone assigned to work on these bugs and generally keep an eye on their driver in nova? How do we get these bugs resolved? Thanks for this call out. I don't think we have anyone specifically assigned to keep an eye on the Ironic Nova driver, we would look at it from time to time or when someone ask us to in the Ironic channel/ML/etc... But that said, I think we need to pay more attention to the bugs in Nova. I've added one item about it to be discussed in the next Ironic meeting[1]. And in the meantime, I will take a look at some of the bugs myself. [1] https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting Thanks to you both for raising this and pushing on this. Maybe we can get a named cross project liaison to bridge the Ironic and Nova meetings. We are working on building a similar pattern for Neutron. It doesn't necessarily mean attending every nova-meeting, just someone to act as an explicit bridge between our two projects? I am open to whatever works though, just hoping we can be more proactive about issues and dependencies that pop up. Thanks, John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Neutron] Cross-project coordination
On 8 May 2015 at 00:24, Brent Eagles beag...@redhat.com wrote: Hi, On Thu, May 07, 2015 at 10:03:30PM +, Sean M. Collins wrote: On Wed, Apr 22, 2015 at 06:41:58PM EDT, Jay Pipes wrote: Agreed. I'm hoping that someone in the Nova community -- note, this does not need to be a Nova core contributor -- can step up to the plate and serve in this critical role. Hi, I've put together a section on the wiki, https://wiki.openstack.org/wiki/CrossProjectLiaisons#Inter-project_Liaisons We still need a Nova liaison to sign up, to help me with the Nova/Neutron cross project effort. If you are interested, please reply and replace the Volunteer Needed sections in the table! I volunteer! I've modified the wiki accordingly. Cheers, Brent Awesome, thank you both! John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Service group foundations and features
On 7 May 2015 at 22:52, Joshua Harlow harlo...@outlook.com wrote: Hi all, In seeing the following: - https://review.openstack.org/#/c/169836/ - https://review.openstack.org/#/c/163274/ - https://review.openstack.org/#/c/138607/ Vilobh and I are starting to come to the conclusion that the service group layers in nova really need to be cleaned up (without adding more features that only work in one driver), or removed or other... Spec[0] has interesting findings on this: A summary/highlights: * The zookeeper service driver in nova has probably been broken for 1 or more releases, due to eventlet attributes that are gone that it via evzookeeper[1] library was using. Evzookeeper only works for eventlet 0.17.1. Please refer to [0] for details. * The memcache service driver really only uses memcache for a tiny piece of the service liveness information (and does a database service table scan to get the list of services). Please refer to [0] for details. * Nova-manage service disable (CLI admin api) does interact with the service group layer for the 'is_up'[3] API (but it also does a database service table scan[4] to get the list of services, so this is inconsistent with the service group driver API 'get_all'[2] view on what is enabled/disabled). Please refer to [9][10] for nova manage service enable disable for details. * Nova service delete (REST api) seems to follow a similar broken pattern (it also avoids calling into the service group layer to delete a service, which means it only works with the database layer[5], and therefore is inconsistent with the service group 'get_all'[2] API). ^^ Doing the above makes both disable/delete agnostic about other backends available that may/might manage service group data for example zookeeper, memcache, redis etc... Please refer [6][7] for details. Ideally the API should follow the model used in [8] so that the extension, admin interface as well as the API interface use the same servicegroup interface which should be *fully* responsible for managing services. Doing so we will have a consistent view of services data, liveness, disabled/enabled and so-on... So with no disrespect to the authors of 169836 and 163274 (or anyone else involved), I am wondering if we can put a request in to figure out how to get the foundation of the service group concepts stabilized (or other...) before adding more features (that only work with the DB layer). What is the path to request some kind of larger coordination effort by the nova folks to fix the service group layers (and the concepts that are not disjoint/don't work across them) before continuing to add features on-top of a 'shakey' foundation? If I could propose something it would probably work out like the following: Step 0: Figure out if the service group API + layer(s) should be maintained/tweaked at all (nova-core decides?) If maintain it: - Have an agreement that nova service extension, admin interface(nova-manage) and API go through a common path for update/delete/read. * This common path should likely be the servicegroup API so as to have a consistent view of data and that also helps nova to add different data-stores (keeping the services data in a DB and getting numerous updates about liveliness every few seconds of N number of compute where N is pretty high can be detrimental to Nova's performance) - At the same time allow 163274 to be worked on (since it fixes a edge-case that was asked about in the initial addition of the delete API in its initial code commit @ https://review.openstack.org/#/c/39998/) - Delay 169836 until the above two/three are fixed (and stabilized); it's down concept (and all other usages of services that are hitting a database mentioned above) will need to go through the same service group foundation that is currently being skipped. Else: - Discard 138607 and start removing the service group code (and just use the DB for all the things). - Allow 163274 and 138607 (since those would be additions on-top of the DB layer that will be preserved). Thoughts? I wonder about this approach: * I think we need to go back and document what we want from the service group concept. * Then we look at the best approach to implement that concept. * Then look at the best way to get to a happy place from where we are now, ** Noting we will need live upgrade for (at least) the most widely used drivers Does that make any sense? Things that pop into my head, include: * The operators have been asking questions like: Should new services not be disabled by default? and Can't my admins tell you that I just killed it? * And from the scheduler point of view, how do we interact with the provider that tells us if something is alive or not? * From the RPC api point of view, do we want to send a cast to something that we know is dead, maybe we want to? Should we wait for calls to timeout, or give up quicker? * Polling the DB kinda sucks,
Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
On 9 May 2015 at 15:02, Mike Bayer mba...@redhat.com wrote: On 5/9/15 6:45 AM, John Garbutt wrote: I am leaning towards us moving to making DB calls with a thread pool and some fast C based library, so we get the 'best' performance. Is that a crazy thing to be thinking? What am I missing here? Thanks, John I'd like to do that but I want the whole Openstack DB API layer in the thread pool, not just the low level DBAPI (Python driver) calls. There's no need for eventlet-style concurrency or even less for async-style concurrency in transactionally-oriented code. Sorry, not sure I get which DB API is which. I was thinking we could dispatch all calls to this API into a thread pool: https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py I guess an alternative is to add this in the objects layer, on top of the rpc dispatch: https://github.com/openstack/nova/blob/master/nova/objects/base.py#L188 But that somehow feels like a layer violation, maybe its not. Is that similar to what you where thinking? Thanks, John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova-docker] Status update
John, Good questions. Remarks in-line from the Magnum perspective. On May 9, 2015, at 2:51 AM, John Garbutt j...@johngarbutt.com wrote: On 1 May 2015 at 16:14, Davanum Srinivas dava...@gmail.com wrote: Anyone still interested in this work? :) * there's a stable/kilo branch now (see http://git.openstack.org/cgit/stackforge/nova-docker/). * CI jobs are running fine against both nova trunk and nova's stable/kilo branch. * there's an updated nova-spec to get code back into nova tree (see https://review.openstack.org/#/c/128753/) To proxy the discussion from the etherpad onto the ML, we need to work out why this lives in nova, given Magnum is the place to do container specific things. To the extent that users want to control Docker containers through the Nova API (without elaborate extensions), I think a stable in-tree nova-docker driver makes complete sense for that. LXC in nova means you can do Magnum containers inside Nova containers, and some folks like the idea of that. You can also use containers just like VMs, for where that makes sense. Yes, we do want the ability to form Bays where the Nodes are Nova instances created form the libvirt-lxc virt driver. In fact, ideally we’d like to be able to use any Nova instance, regardless of what virt driver was used to produce it. That logic extends to use of the nova-docker virt driver as well. In that case we would have a Nova instance that’s a docker container inside which we have members of a container cluster like Kubernetes or Swarm that produce containers within the nova container instances. I refer to this arrangement as “nested containers”. See more about this below. Practically speaking, we may need to initially focus on a very short list of tested virt-drivers. Over time, I’d like to see that list expand to point where we can claim that Magnum is completely virt driver agnostic. Now whats the reason for adding the Docker driver, given Nova is considering container specific APIs out of scope, and expecting Magnum to own that kind of thing. I do think nova-docker should find it’s way into the Nova tree. This makes containers more accessible in OpenStack, and appropriate for use cases where users want to treat containers like they treat virtual machines. On the subject of extending the Nova API to accommodate special use cases of containers that are beyond the scope of the Nova API, I think we should resist that, and focus those container-specific efforts in Magnum. That way, cloud operators can choose whether to use Nova or Magnum for their container use cases depending on the range of features they desire from the API. This approach should also result in less overlap of efforts. I will also mention that it’s natural to be allergic to the idea of nested virtualization. We all know that creating multiple levels of hardware virtualization leads to bad performance outcomes. However, nested containers do not carry that same drawback, because the actual overhead of a Linux cgroup and Kernel Namespeaces are much lighter than a hardware virtualization. There are cases where having a container-in-container setup gives compelling benefits. That’s why I’ll argue vigorously for both Nova and Magnum to be able to produce container instances both at the machine level, and allow Magnum to produce nested containers” to produce better workload consolidation density. in a setup with no hypervisors at all. To sum up, I strongly support merging in nova-docker, with the caveat that it operates within the existing Nova API (with few minor exceptions). For features that require API features that are truly container specific, we should land those in Magnum, and keep the Nova API scoped to operations that are appropriate for “all instance types. Adrian Thanks, John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
On 5/9/15 6:45 AM, John Garbutt wrote: I am leaning towards us moving to making DB calls with a thread pool and some fast C based library, so we get the 'best' performance. Is that a crazy thing to be thinking? What am I missing here? Thanks, John I'd like to do that but I want the whole Openstack DB API layer in the thread pool, not just the low level DBAPI (Python driver) calls. There's no need for eventlet-style concurrency or even less for async-style concurrency in transactionally-oriented code. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] global-requirements not co-installable
On 9 May 2015 at 23:42, Jeremy Stanley fu...@yuggoth.org wrote: On 2015-05-09 06:46:59 +1200 (+1200), Robert Collins wrote: As I read it, we've got some tooling that isn't PEP-440 compatible (https://www.python.org/dev/peps/pep-0440/#compatible-release defines ~=) and as such we had to rollback the intended use of that. As long as we identify and fix those tools, we should be fine. Did anyone involved with that situation create a bug we can use to track this? I don't think it has anything to do with the choice of cap-or-not. We had (have) virtualenv versions in use in automation which bundle a pip and setuptools too old to support this... lingering fallout This is my understanding: We explicitly install pip latest in our jobs (via get-pip, retrieved over https). Virtualenv shouldn't be version sensitive at all to this since its not evaluating versions of anything. setuptools may end up having a lower bound for environment markers (since we don't want to trigger easy_install when pip rightly ignores a different platform dependency). The virtualenv -pypy thing is different, because it's dealing with pypy quirks around some stdlib module, from memory. from the scramble around pinning these bits of the toolchain at the end of last year while we tried to keep everything running. Clark has proposed https://review.openstack.org/181558 which we can try and then quickly revert if too much explodes. Cool, lets do that. However it's worth noting, as soon as we start adding ~= into requirements files we're basically saying we no longer support installing any of those packages with pip6 or setuptools8 (or with virtualenv12 without an implicit upgrade step in the resulting env). The supporting versions of these tools/libs aren't even 6 months old yet. If we want to fix the plumbing, we should expect to run the latest releases of pip, wheel, setuptools during the fixing period, until all the relevant bugs we have are fixed; the age of the releases doesn't mean a lot. For folk running tooling matching CI's tooling, they need to match minimum versions too. For folk running distro packaging based tooling, they may need to match minimum versions in the package build toolchain, but at runtime pip and virtualenv and wheel and setuptools will all be irrelevant. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit downtime and upgrade on Saturday 2015-05-09 at 1600 UTC
cor...@inaugust.com (James E. Blair) writes: On Saturday, May 9 at 16:00 UTC Gerrit will be unavailable for about 4 hours while we upgrade to the latest release of Gerrit: version 2.10. We are currently running Gerrit 2.8 so this is an upgrade across two major releases of Gerrit. The release notes for both versions are here: https://gerrit-documentation.storage.googleapis.com/ReleaseNotes/ReleaseNotes-2.10.html https://gerrit-documentation.storage.googleapis.com/ReleaseNotes/ReleaseNotes-2.9.html If you have any questions about the upgrade, please feel free to reply here or contact us in #openstack-infra on Freenode. This work was successfully completed around 18:30 UTC. Thanks to Khai Do who prepared and tested the changes for the upgrade, and thanks to Khai Do, Jeremy Stanley, Clark Boylan, and Monty Taylor for assisting in the performance of the upgrade. If you encounter any problems, please let us know here or in #openstack-infra on Freenode. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] global-requirements not co-installable
On 2015-05-10 07:09:38 +1200 (+1200), Robert Collins wrote: This is my understanding: We explicitly install pip latest in our jobs (via get-pip, retrieved over https). Virtualenv shouldn't be version sensitive at all to this since its not evaluating versions of anything. My point was that, because we're running with virtualenv 1.11.4 currently we end up using pip 1.5.4 which comes bundled within it, unless the venv itself is explicitly upgraded after creation. It doesn't use the system-installed pip version we have from get-pip. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] global-requirements not co-installable
On 10 May 2015 at 07:19, Jeremy Stanley fu...@yuggoth.org wrote: On 2015-05-10 07:09:38 +1200 (+1200), Robert Collins wrote: This is my understanding: We explicitly install pip latest in our jobs (via get-pip, retrieved over https). Virtualenv shouldn't be version sensitive at all to this since its not evaluating versions of anything. My point was that, because we're running with virtualenv 1.11.4 currently we end up using pip 1.5.4 which comes bundled within it, unless the venv itself is explicitly upgraded after creation. It doesn't use the system-installed pip version we have from get-pip. Ok, so thats a bug no matter what virtualenv version we're using, because we assume latest pip in all of our other design and impl work :/. We should do the following immediately we create a virtualenv anywhere in our infra: get-pip.py pip install -U wheel setuptools So that we're not running into old-stuff surprises, and so that the pip 7 wheel autobuilding can work. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] On dynamic policy, role hierarchies/groups/sets etc. - Role Assignment
Hi Tim I was implying that the addRole operation would not be used or needed in the federation case, because all user roles are initially created by IdPs and then by attribute mappings. I was not saying anything about the various admin roles that might exist because as I understand it, there is no limitation on the number of roles that can be defined in OpenStack. regards David On 08/05/2015 15:52, Tim Hinrichs wrote: Hi David, See below. On 5/7/15, 1:01 AM, David Chadwick d.w.chadw...@kent.ac.uk wrote: Hi Tim On 06/05/2015 21:53, Tim Hinrichs wrote: I wondered if we could properly protect the API call for adding a new Role using the current mechanism. So I came up with a simple example. Suppose we want to write policy about the API call: addRole(user, role-name). If we¹re hosting both Pepsi and Coke, we want to write a policy that says that only someone in the Pepsi admin role can change roles for Pepsi users (likewise for Coke). We¹d want to write something likeŠ addRole(user, role) is permitted for caller if caller belongs to the Pepsi-admin role and user belongs to the Pepsi role The policy engine knows if ³caller belongs to the Pepsi-admin role² because that¹s part of the token. But the policy engine doesn¹t know if ³user belongs to the Pepsi role² because user is just an argument to the API call, so we don¹t have role info about user. This helps me understand *why* we can¹t handle the multi-customer use case right now: the policy engine doesn¹t have all the info it needs. But otherwise, it seems, we could handle the multi-customer use-case using mechanism that already exists. Are there other examples where they can¹t write policy because the engine doesn¹t have enough info? Your simple example does not work in the federated case. This is because role and attribute assignments are not done by Keystone, or by any part of Openstack, but by a remote IDP. It is assumed that the administrator of this remote IDP knows who his users are, and will assign the correct attributes to them. However, these are not necessarily OpenStack roles (they most certainly wont be). Therefore, we have built a perfectly good mechanism into Keystone, to ensure that the users from any IDP (Coke, Pepsi or Virgin Cola etc.) get the right Keystone/Openstack role(s), and this is via attibute mapping. When the mapping takes place, the user is in the process of logging in, therefore Keystone knows the attributes of the user (assigned by the IDP) and can therefore know which Openstack role to assign to him/her. I understand the idea of mapping attributes from a remote IDP to OpenStack/Keystone roles. But I don¹t understand the impact on my example. In my example, the policy statement fails to work for one of 2 reasons: 1. there¹s no such thing as a Pepsi-admin role 2. The policy engine can¹t check if ³user belongs to Pepsi The policy statement fails to work because of (2) for sure. But are you saying it also fails to work because of (1) in the federated case? I would have thought that the Keystone roles used to represent the Pepsi IDP attributes would be separate from the Keystone roles used to represent Coke IDP attributes, and therefore there¹s be some role corresponding to Pepsi-admin and Coke-admin. Sorry if this is obvious. Tim I hope this helps. regards David __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
