Re: [openstack-dev] [Ironic][Neutron] - Integration with neutron using external attachment point
We've been experimenting some with how to use Neutron with Ironic here at Rackspace. Our very experimental code: https://github.com/rackerlabs/ironic-neutron-plugin Our objective is the same as what you're describing, to allow Nova servers backed by Ironic to attach to arbitrary Neutron networks. We're initially targeting VLAN-based networks only, but eventually want to do VXLAN from the top-of-rack switches, controlled via an SDN controller. Our approach is a little different than what you're describing though. Our objective is to modify the existing Nova -> Neutron interaction as little as possible, which means approaching the problem by thinking "how would an L2 agent do this?". The workflow looks something like: 1. Nova calls Neutron to create a virtual "port". Because this happens _before_ Nova touches the virt driver, the port is at this point identical to one created for a virtual server. 2. Nova executes the "spawn" method of the Ironic virt driver, which makes some calls to Ironic. 3. Inside Ironic, we know about the physical switch ports that the selected Node is connected to. This information is discovered early-on using LLDP and stored in the Ironic database. 4. We actually need the node to remain on an internal provisioning VLAN for most of the provisioning process, but once we're done with on-host work we turn the server off. 5. Ironic deletes a Neutron port that was created at bootstrap time to trunk the physical switch ports for provisioning. 6. Ironic updates each of the customer's Neutron ports with information about its physical switch port. 6. Our Neutron extension configures the switches accordingly. 7. Then Ironic brings the server back up. The destroy process basically does the reverse. Ironic removes the physical switch mapping from the Neutron ports, re-creates an internal trunked port, does some work to tear down the server, then passes control back to Nova. At that point Nova can do what it wants with the Neutron ports. Hypothetically that could include allocating them to a different Ironic Node, etc, although in practice it just deletes them. Again, this is all very experimental in nature, but it seems to work fairly well for the use-cases we've considered. We'd love to find a way to collaborate with others working on similar problems. Thanks, Russell On Tue, May 20, 2014 at 7:17 AM, Akihiro Motoki wrote: > # Added [Neutron] tag as well. > > Hi Igor, > > Thanks for the comment. We already know them as I commented > in the Summit session and ML2 weekly meeting. > Kevin's blueprint now covers Ironic integration and layer2 network gateway > and I believe "campus-network" blueprint will be covered. > > We think the work can be split into generic API definition and > implementations > (including ML2). In "external attachment point" blueprint review, API > and generic topics are mainly discussed so far and the detail > implementation is not discussed > so much yet. ML2 implementation detail can be discussed later > (separately or as a part of the blueprint review). > > I am not sure what changes proposed in Blueprint [1]. > AFAIK SDN/OpenFlow controller based approach can support this, > but how can we archive this in the existing open source implementation. > I am also interested in the ML2 implementation detail. > > Anyway more input will be appreciated. > > Thanks, > Akihiro > > On Tue, May 20, 2014 at 7:13 PM, Igor Cardoso wrote: > > Hello Kevin. > > There is a similar Neutron blueprint [1], originally meant for Havana but > > now aiming for Juno. > > I would be happy to join efforts with you regarding our blueprints. > > See also: [2]. > > > > [1] https://blueprints.launchpad.net/neutron/+spec/ml2-external-port > > [2] https://blueprints.launchpad.net/neutron/+spec/campus-network > > > > > > On 19 May 2014 23:52, Kevin Benton wrote: > >> > >> Hello, > >> > >> I am working on an extension for neutron to allow external attachment > >> point information to be stored and used by backend plugins/drivers to > place > >> switch ports into neutron networks[1]. > >> > >> One of the primary use cases is to integrate ironic with neutron. The > >> basic workflow is that ironic will create the external attachment points > >> when servers are initially installed. This step could either be > automated > >> (extract switch-ID and port number of LLDP message) or it could be > manually > >> performed by an admin who notes the ports a server is plugged into. > >> > >> Then when an instance is chosen for assignment and the neutron port > needs > >> to be created, the creation request would reference the corresponding > >> attachment ID and neutron would configure the physical switch port to > place > >> the port on the appropriate neutron network. > >> > >> If this workflow won't work for Ironic, please respond to this email or > >> leave comments on the blueprint review. > >> > >> 1. https://review.openstack.org/#/c/87825/ > >> > >> > >> Thanks > >> -- > >> Kevin Benton > >> > >>
Re: [openstack-dev] [Ironic] Moving to a formal design process
+1 to this process, and I think the template is pretty reasonable. One thing to call out maybe, is are there any known failure scenarios? If so, why shouldn't they block the spec? On Mon, May 19, 2014 at 5:51 PM, Devananda van der Veen < devananda@gmail.com> wrote: > Added - > > https://github.com/devananda/ironic-specs/commit/7f34f353332ad5b26830dadc8c9f870df399feb7 > > > On Sun, May 18, 2014 at 6:13 PM, Robert Collins > wrote: > >> I'd like to suggest two things. >> >> Firstly a section on scale (as opposed to performance). >> >> Secondly, I'd like to see additional hard requirements that will be >> added to drivers called out (e.g. a 'Driver Impact' section). >> >> -Rob >> >> On 19 May 2014 10:03, Devananda van der Veen >> wrote: >> > Hi all, >> > >> > As with several other projects, and as discussed at the summit, Ironic >> is >> > moving to a formal / specs-based design process. The reasons for this >> have >> > been well summarized in previous email threads in other projects [*], >> but in >> > short, it's because, until now, nearly all our blueprints lacked a >> design >> > specification which could be compared to the proposed code, resulting >> in the >> > code-review also being a design-review. This week, I will be resetting >> the >> > "Definition" status of all blueprints to "New", and require everything >> to go >> > through a specs review process -- yes, even the ones that were >> previously >> > approved. >> > >> > I've proposed the creation of the openstack/ironic-specs repo here: >> > https://review.openstack.org/#/c/94113/ >> > >> > And put up an initial version on github to start the process: >> > https://github.com/devananda/ironic-specs >> > >> https://github.com/devananda/ironic-specs/blob/master/specs/template.rst >> > >> > I've begun sketching out specs proposals for some of the >> essential-for-juno >> > items, which can be found in the above repo for now, and will be >> proposed >> > for review once the openstack/ironic-specs repository is created. >> > >> > For what it's worth, I based this on the Nova specs repo, with some >> > customization geared towards Ironic (eg, I removed "notifications" >> section >> > and added some comments regarding hardware). At this point, I'd like >> > feedback from other core reviewers and folks familiar with the specs >> > process. Please think about the types of architectural changes you look >> for >> > during code reviews and make sure the specs template addresses them, and >> > that they apply to Ironic. >> > >> > I will focus on creating and landing the specs for items essential for >> > graduation first and then prioritize review of additional feature specs >> > based on the community feedback we received at the Juno Design Summit. I >> > will send another email soliciting the community members and vendors to >> > propose specs for the features they are working on once the initial >> work on >> > the repo is complete (hopefully before end-of-week). >> > >> > Regards, >> > Devananda >> > >> > >> > [*] eg.: >> > >> http://lists.openstack.org/pipermail/openstack-dev/2014-April/032753.html >> > >> > ___ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> >> >> -- >> Robert Collins >> Distinguished Technologist >> HP Converged Cloud >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] Should we adopt a blueprint design process
Completely agree. We're spending too much time discussing features after they're implemented, which makes contribution more difficult for everyone. Forcing an explicit design+review process, using the same tools as we use for coding+review seems like a great idea. If it doesn't work we can iterate. On Thu, Apr 17, 2014 at 11:01 AM, Kyle Mestery wrote: > On Thu, Apr 17, 2014 at 12:11 PM, Devananda van der Veen > wrote: > > Hi all, > > > > The discussion of blueprint review has come up recently for several > reasons, > > not the least of which is that I haven't yet reviewed many of the > blueprints > > that have been filed recently. > > > > My biggest issue with launchpad blueprints is that they do not provide a > > usable interface for design iteration prior to writing code. Between the > > "whiteboard" section, wikis, and etherpads, we have muddled through a few > > designs (namely cinder and ceilometer integration) with accuracy, but the > > vast majority of BPs are basically reviewed after they're implemented. > This > > seems to be a widespread objection to launchpad blueprints within the > > OpenStack community, which others are trying to solve. Having now looked > at > > what Nova is doing with the nova-specs repo, and considering that > TripleO is > > also moving to that format for blueprint submission, and considering > that we > > have a very good "review things in gerrit" culture in the Ironic > community > > already, I think it would be a very positive change. > > > > For reference, here is the Nova discussion thread: > > > http://lists.openstack.org/pipermail/openstack-dev/2014-March/029232.html > > > > and the specs repo BP template: > > https://github.com/openstack/nova-specs/blob/master/specs/template.rst > > > > So, I would like us to begin using this development process over the > course > > of Juno. We have a lot of BPs up right now that are light on details, > and, > > rather than iterate on each of them in launchpad, I would like to propose > > that: > > * we create an ironic-specs repo, based on Nova's format, before the > summit > > * I will begin reviewing BPs leading up to the summit, focusing on > features > > that were originally targeted to Icehouse and didn't make it, or are > > obviously achievable for J1 > > * we'll probably discuss blueprints and milestones at the summit, and > will > > probably adjust targets > > * after the summit, for any BP not targeted to J1, we require blueprint > > proposals to go through the spec review process before merging any > > associated code. > > > > Cores and interested parties, please reply to this thread with your > > opinions. > > > I think this is a great idea Devananda. The Neutron community has > moved to this model for Juno as well, and people have been very > positive so far. > > Thanks, > Kyle > > > -- > > Devananda > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] Merging the agent driver
All, If anyone has a chance to do some review, we really want to get this one merged: https://review.openstack.org/#/c/81919/ This will clear the way for us to work on getting the driver reviewed, followed by a series of smaller patches. Thanks, Russell On Tue, Apr 8, 2014 at 4:58 PM, Jim Rollenhagen wrote: > Hi all, > > As Deva requested, our team put up a merge request for the IPA driver ( > https://review.openstack.org/#/c/84795/) as soon as Juno opened. We’ve > continued to update this patch and iterate on the agent model. We are doing > our best not to get too far ahead of master, but we also want to get a > working prototype done ASAP, specifically before Atlanta. > > This merge request needs reviews, but also depends on two other reviews: > - https://review.openstack.org/#/c/81391/ > - https://review.openstack.org/#/c/81919/ > > The reason I would like to get these patches moving quickly is because we > have some other patches depending on them: > - https://review.openstack.org/#/c/85228/ > - https://review.openstack.org/#/c/86173/ > - https://review.openstack.org/#/c/86141/ > > And we’re developing more as we speak. > > So, that said - what can we do to help get these patches through faster? > Whether that means our team reviewing Ironic things to help out Ironic > cores, or answering questions about the patches that are up, we’re willing > to do it to get these through faster. > > // jim > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] Nodeless Vendor Passthru API
On Tue, Mar 25, 2014 at 6:56 AM, Lucas Alvares Gomes wrote: > > Hi Russell, > > Ironic allows drivers to expose a "vendor passthru" API on a Node. This >> basically serves two purposes: >> >> 1. Allows drivers to expose functionality that hasn't yet been >> standardized in the Ironic API. For example, the Seamicro driver exposes >> "attach_volume", "set_boot_device" and "set_node_vlan_id" passthru methods. >> 2. Vendor passthru is also used by the PXE deploy driver as an internal >> RPC callback mechanism. The deploy ramdisk makes calls to the passthru API >> to signal for a deployment to continue once a server has booted. >> >> For the purposes of this discussion I want to focus on case #2. Case #1 >> is certainly worth a separate discussion - we started this in >> #openstack-ironic on Friday. >> >> In the new agent we are working on, we want to be able to look up what >> node the agent is running on, and eventually to be able to register a new >> node automatically. We will perform an inventory of the server and submit >> that to Ironic, where it can be used to map the agent to an existing Node >> or to create a new one. Once the agent knows what node it is on, it will >> check in with a passthru API much like that used by the PXE driver - in >> some configurations this might trigger an immediate "continue" of an >> ongoing deploy, in others it might simply register the agent as available >> for new deploys in the future. >> > > Maybe another way to look up what node the agent is running on would be by > looking at the MAC address of that node, having it on hand you could then > GET /ports/detail and find which port has that MAC associated with it, > after you find the port you can look at the node_uuid field which holds the > UUID of the node which that port belongs to (All ports have a node_uuid, > it's mandatory). So, right now you would need to list all the ports and > find that MAC address there, but I got a review up that might help you with > it by allowing you to get a port using its address as input: > https://review.openstack.org/#/c/82773/ (GET /ports/detail?address=). > > What you think? > Right, we discussed this possibility as well. Internally that's actually how the first iteration of our lookup call will work (mapping MAC addresses to ports to nodes). This can definitely be made to work, but in my mind it has a few limitations: 1. It limits how we can do lookups. In the future I'd like to be able to consider serial numbers, hardware profiles, etc when trying to map an agent to a node. Needing to expose an API for each of these is going to be infeasible. 2. It limits how we do enrollment. 3. It forces us to expose more of the API to agents. Items 1 and 2 seem like things that someone deploying Ironic is especially likely to want to customize. For example, I might have some business process around replacing NICs where I want to customize how a server is uniquely identified (or even hit an external source to do it), or I might want want to override enrollment to hook into Fuel. While we can probably find a way to solve our current problem, how can we generically solve the need for an agent to talk to a driver (either in order to centralize orchestration, or because access to the database is needed) without needing a Node UUID? Thanks, Russell ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ironic] Nodeless Vendor Passthru API
All, Ironic allows drivers to expose a "vendor passthru" API on a Node. This basically serves two purposes: 1. Allows drivers to expose functionality that hasn't yet been standardized in the Ironic API. For example, the Seamicro driver exposes "attach_volume", "set_boot_device" and "set_node_vlan_id" passthru methods. 2. Vendor passthru is also used by the PXE deploy driver as an internal RPC callback mechanism. The deploy ramdisk makes calls to the passthru API to signal for a deployment to continue once a server has booted. For the purposes of this discussion I want to focus on case #2. Case #1 is certainly worth a separate discussion - we started this in #openstack-ironic on Friday. In the new agent we are working on, we want to be able to look up what node the agent is running on, and eventually to be able to register a new node automatically. We will perform an inventory of the server and submit that to Ironic, where it can be used to map the agent to an existing Node or to create a new one. Once the agent knows what node it is on, it will check in with a passthru API much like that used by the PXE driver - in some configurations this might trigger an immediate "continue" of an ongoing deploy, in others it might simply register the agent as available for new deploys in the future. The point here is that we need a way for the agent driver to expose a top-level "lookup" API, which doesn't require a Node UUID in the URL. I've got a review (https://review.openstack.org/#/c/81919/) up which explores one possible implementation of this. It basically routes POSTs to /drivers//vendor_passthru/ to a new method on the vendor interface. Importantly, I don't believe that this is a useful way for vendors to implement new consumer-facing functionality. If we decide to take this approach, we should reject drivers try to do so. It is intended *only* for internal communications with deploy agents. Another possibility is that we could create a new API service intended explicitly to serve use case #2 described above, which doesn't include most of the existing public paths. In our environment I expect us to allow agents whitelisted access to only two specific paths (lookup and checkin), but this might be a better way to achieve that. Thoughts? Thanks, Russell ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] A ramdisk agent
Thanks for putting that info together! I'm not sure exactly what order things need to happen in, but Jay (JayF) is working on the infra bits of getting a repository and CI, and Jim (jroll) is getting the Pecan+WSME part done. Hopefully we'll have it all ready by Monday. On Fri, Mar 7, 2014 at 12:53 PM, Vladimir Kozhukalov < vkozhuka...@mirantis.com> wrote: > Russell, > > Great to hear you are going to move towards Pecan+WSME. Yesterday I had a > look at teeth projects. Next few days I am going to start contributing. > First of all, I think, we need to arrange all that stuff about pluggable > architecture. I've created a wiki page about Ironic python agent > https://wiki.openstack.org/wiki/Ironic-python-agent. > > And the question about contributing. Have you managed to send pull request > to openstack-infra in order to move this project into > github.com/stackforge? Or we are supposed to arrange everything (werkzeug > -> Pecan/WSME, architectural questions) before we move this agent to > stackforge? > > > > > > Vladimir Kozhukalov > > > On Fri, Mar 7, 2014 at 8:53 PM, Russell Haering > wrote: > >> Vladmir, >> >> Hey, I'm on the team working on this agent, let me offer a little >> history. We were working on a system of our own for managing bare metal >> gear which we were calling "Teeth". The project was mostly composed of: >> >> 1. teeth-agent: an on-host provisioning agent >> 2. teeth-overlord: a centralized automation mechanism >> >> Plus a few other libraries (including teeth-rest, which contains some >> common code we factored out of the agent/overlord). >> >> A few weeks back we decided to shift our focus to using Ironic. At this >> point we have effectively abandoned teeth-overlord, and are instead >> focusing on upstream Ironic development, continued agent development and >> building an Ironic driver capable of talking to our agent. >> >> Over the last few days we've been removing non-OS-approved dependencies >> from our agent: I think teeth-rest (and werkzeug, which it depends on) will >> be the last to go when we replace it with Pecan+WSME sometime in the next >> few days. >> >> Thanks, >> Russell >> >> >> On Fri, Mar 7, 2014 at 8:26 AM, Vladimir Kozhukalov < >> vkozhuka...@mirantis.com> wrote: >> >>> As far as I understand, there are 4 projects which are connected with >>> this topic. Another two projects which were not mentioned by Devananda are >>> https://github.com/rackerlabs/teeth-rest >>> https://github.com/rackerlabs/teeth-overlord >>> >>> Vladimir Kozhukalov >>> >>> >>> On Fri, Mar 7, 2014 at 4:41 AM, Devananda van der Veen < >>> devananda@gmail.com> wrote: >>> >>>> All, >>>> >>>> The Ironic team has been discussing the need for a "deploy agent" since >>>> well before the last summit -- we even laid out a few blueprints along >>>> those lines. That work was deferred and we have been using the same deploy >>>> ramdisk that nova-baremetal used, and we will continue to use that ramdisk >>>> for the PXE driver in the Icehouse release. >>>> >>>> That being the case, at the sprint this week, a team from Rackspace >>>> shared work they have been doing to create a more featureful hardware agent >>>> and an Ironic driver which utilizes that agent. Early drafts of that work >>>> can be found here: >>>> >>>> https://github.com/rackerlabs/teeth-agent >>>> https://github.com/rackerlabs/ironic-teeth-driver >>>> >>>> I've updated the original blueprint and assigned it to Josh. For >>>> reference: >>>> >>>> https://blueprints.launchpad.net/ironic/+spec/utility-ramdisk >>>> >>>> I believe this agent falls within the scope of the baremetal >>>> provisioning program, and welcome their contributions and collaboration on >>>> this. To that effect, I have suggested that the code be moved to a new >>>> OpenStack project named "openstack/ironic-python-agent". This would follow >>>> an independent release cycle, and reuse some components of tripleo >>>> (os-*-config). To keep the collaborative momentup up, I would like this >>>> work to be done now (after all, it's not part of the Ironic repo or >>>> release). The new driver which will interface with that agent will need to >>>> stay on github -- or in a gerrit feature br
Re: [openstack-dev] [Ironic] A ramdisk agent
Vladmir, Hey, I'm on the team working on this agent, let me offer a little history. We were working on a system of our own for managing bare metal gear which we were calling "Teeth". The project was mostly composed of: 1. teeth-agent: an on-host provisioning agent 2. teeth-overlord: a centralized automation mechanism Plus a few other libraries (including teeth-rest, which contains some common code we factored out of the agent/overlord). A few weeks back we decided to shift our focus to using Ironic. At this point we have effectively abandoned teeth-overlord, and are instead focusing on upstream Ironic development, continued agent development and building an Ironic driver capable of talking to our agent. Over the last few days we've been removing non-OS-approved dependencies from our agent: I think teeth-rest (and werkzeug, which it depends on) will be the last to go when we replace it with Pecan+WSME sometime in the next few days. Thanks, Russell On Fri, Mar 7, 2014 at 8:26 AM, Vladimir Kozhukalov < vkozhuka...@mirantis.com> wrote: > As far as I understand, there are 4 projects which are connected with this > topic. Another two projects which were not mentioned by Devananda are > https://github.com/rackerlabs/teeth-rest > https://github.com/rackerlabs/teeth-overlord > > Vladimir Kozhukalov > > > On Fri, Mar 7, 2014 at 4:41 AM, Devananda van der Veen < > devananda@gmail.com> wrote: > >> All, >> >> The Ironic team has been discussing the need for a "deploy agent" since >> well before the last summit -- we even laid out a few blueprints along >> those lines. That work was deferred and we have been using the same deploy >> ramdisk that nova-baremetal used, and we will continue to use that ramdisk >> for the PXE driver in the Icehouse release. >> >> That being the case, at the sprint this week, a team from Rackspace >> shared work they have been doing to create a more featureful hardware agent >> and an Ironic driver which utilizes that agent. Early drafts of that work >> can be found here: >> >> https://github.com/rackerlabs/teeth-agent >> https://github.com/rackerlabs/ironic-teeth-driver >> >> I've updated the original blueprint and assigned it to Josh. For >> reference: >> >> https://blueprints.launchpad.net/ironic/+spec/utility-ramdisk >> >> I believe this agent falls within the scope of the baremetal provisioning >> program, and welcome their contributions and collaboration on this. To that >> effect, I have suggested that the code be moved to a new OpenStack project >> named "openstack/ironic-python-agent". This would follow an independent >> release cycle, and reuse some components of tripleo (os-*-config). To keep >> the collaborative momentup up, I would like this work to be done now (after >> all, it's not part of the Ironic repo or release). The new driver which >> will interface with that agent will need to stay on github -- or in a >> gerrit feature branch -- until Juno opens, at which point it should be >> proposed to Ironic. >> >> The agent architecture we discussed is roughly: >> - a pluggable JSON transport layer by which the Ironic driver will pass >> information to the ramdisk. Their initial implementation is a REST API. >> - a collection of hardware-specific utilities (python modules, bash >> scripts, what ever) which take JSON as input and perform specific actions >> (whether gathering data about the hardware or applying changes to it). >> - and an agent which routes the incoming JSON to the appropriate utility, >> and routes the response back via the transport layer. >> >> >> -Devananda >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
