Re: [openstack-dev] [Ironic] Proposal to add a new repository
On 06/23/2015 02:11 AM, Devananda van der Veen wrote: Oh - one more thing. If ahc-tools depends on data gathered by enovance/hardware, then I'm not sure it makes sense to import one to openstack/ without the other. Maybe. We'll chat with our folks about it. -Deva On Mon, Jun 22, 2015 at 5:08 PM Devananda van der Veen devananda@gmail.com mailto:devananda@gmail.com wrote: I'm On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge tr...@redhat.com mailto:tr...@redhat.com wrote: On 06/22/2015 10:40 AM, Dmitry Tantsur wrote: On 06/22/2015 04:19 PM, Devananda van der Veen wrote: Hi John, Thanks for the excellent summary! I found it very helpful to get caught up. I'd like to make sure I understand the direction ahc is going. A couple questions... Thanks for your interest. Let me add my $0.5 :) I see that ahc is storing its information in swift. That's clever, but if Ironic provided a blob store for each node, would that be better? If the blob is large enough, this would be better. Originally we stored the data in the extra column of the Ironic db, but that proved disastrous: https://bugs.launchpad.net/ironic-inspector/+bug/1461252 We discussed adding a search API to Ironic at the Vancouver summit, though no work has been done on that yet, afaik. If ahc is going to grow a REST API for searching for nodes based on specific criteria that it discovered, could/should we combine these within Ironic's API? I think John meant having API to replace scripts, so I guess search won't help. When we're talking about advanced matching, we're talking about the following: 1. We have a ramdisk tool (based on [8]) to get some insane of facts from withing the ramdisk (say, 1000 of them) 2. We have an Inspector plugin to put them all in Swift (or Ironic blob storage as above) 3. We have config files (aka rules) written in special JSON-alike DSL to do matching (one of the weak points is that these are files - I'd like API endpoint to accept these rules instead). 4. We have a script to run this DSL and get some output (match/not match + some matched variables - similar to what regexps do). As I understood it John want the latter to become an API endpoint, accepting rules (and maybe node UUIDs) and outputting some result. Not sure about benchmarking here, but again, it's probably an API endpoint that accepts some minimal expectations, and puts failed nodes to maintenance mode, if they fail to comply (again, that's how I understood it). It's not hard to make these API endpoints part of Inspector, but it's somewhat undesirable to have them optional... From a service coupling perspective, I like the approach that ahc is optional, and also that Ironic-inspector is optional, because this keeps the simple use-case for Ironic, well, simple! That said, this seems more like a configuration setting (should inspector do extra things?) than an entirely separate service, and separating them might be unnecessarily complicated. We keep thinking about it as well. After all, right now it's just a couple of utilities. There are 2 more concerns that initially made me pull out this code: 1. ahc-tools currently depends on the library [8], which I wish would be developed much more openly 2. it's cool that inspector is pluggable, but it has its cost: there's a poor feedback loop from inspector processing plugins to a user - like with all highly asynchronous code 3. it's also not possible (at least for now) to request a set of processing plugins when starting introspection via inspector. We solved the latter 2 problems by moving code to scripts. So now Inspector only puts some data to Swift, and scripts can do everything else. So now we've left with 1. dependency on hardware library 2. not very stable interface, much less stable than one of Inspector We still wonder how to solve these 2 without creating one more repository. Any ideas are welcome :) Oh - good point. There's some neat looking functionality in enovance/hardware repository, but yea,
Re: [openstack-dev] [Ironic] Proposal to add a new repository
On 06/22/2015 10:40 AM, Dmitry Tantsur wrote: On 06/22/2015 04:19 PM, Devananda van der Veen wrote: Hi John, Thanks for the excellent summary! I found it very helpful to get caught up. I'd like to make sure I understand the direction ahc is going. A couple questions... Thanks for your interest. Let me add my $0.5 :) I see that ahc is storing its information in swift. That's clever, but if Ironic provided a blob store for each node, would that be better? If the blob is large enough, this would be better. Originally we stored the data in the extra column of the Ironic db, but that proved disastrous: https://bugs.launchpad.net/ironic-inspector/+bug/1461252 We discussed adding a search API to Ironic at the Vancouver summit, though no work has been done on that yet, afaik. If ahc is going to grow a REST API for searching for nodes based on specific criteria that it discovered, could/should we combine these within Ironic's API? I think John meant having API to replace scripts, so I guess search won't help. When we're talking about advanced matching, we're talking about the following: 1. We have a ramdisk tool (based on [8]) to get some insane of facts from withing the ramdisk (say, 1000 of them) 2. We have an Inspector plugin to put them all in Swift (or Ironic blob storage as above) 3. We have config files (aka rules) written in special JSON-alike DSL to do matching (one of the weak points is that these are files - I'd like API endpoint to accept these rules instead). 4. We have a script to run this DSL and get some output (match/not match + some matched variables - similar to what regexps do). As I understood it John want the latter to become an API endpoint, accepting rules (and maybe node UUIDs) and outputting some result. Not sure about benchmarking here, but again, it's probably an API endpoint that accepts some minimal expectations, and puts failed nodes to maintenance mode, if they fail to comply (again, that's how I understood it). It's not hard to make these API endpoints part of Inspector, but it's somewhat undesirable to have them optional... From a service coupling perspective, I like the approach that ahc is optional, and also that Ironic-inspector is optional, because this keeps the simple use-case for Ironic, well, simple! That said, this seems more like a configuration setting (should inspector do extra things?) than an entirely separate service, and separating them might be unnecessarily complicated. We keep thinking about it as well. After all, right now it's just a couple of utilities. There are 2 more concerns that initially made me pull out this code: 1. ahc-tools currently depends on the library [8], which I wish would be developed much more openly 2. it's cool that inspector is pluggable, but it has its cost: there's a poor feedback loop from inspector processing plugins to a user - like with all highly asynchronous code 3. it's also not possible (at least for now) to request a set of processing plugins when starting introspection via inspector. We solved the latter 2 problems by moving code to scripts. So now Inspector only puts some data to Swift, and scripts can do everything else. So now we've left with 1. dependency on hardware library 2. not very stable interface, much less stable than one of Inspector We still wonder how to solve these 2 without creating one more repository. Any ideas are welcome :) It is a goal of mine to solve issue 1 incrementally over time. Either by improving the library (both in function and in openness), or by slowly moving the implementation. That does not seem impossible to do within the inspector tree. However, issue 2 is a fact. We currently have scripts, and we want to have a REST API. I do not see a transition between the two that does not involve a large amount of churn. I am not sure how to solve issue 2 without a separate repository. I do think there is a logical separation of concerns though, so we may not need to completely merge the two in the future. Inspector collects data, and ahc-tools (or whatever it is eventually named) is used to act on the data. It sounds like this is the direction you'd like to go, and you took the current approach for expediency. If so, I'd like us to discuss a path to merge the functionality as it matures, and decide whether a separate repository is the right way to go long term. Thanks much, Devananda On Mon, Jun 22, 2015, 05:40 John Trowbridge tr...@redhat.com mailto:tr...@redhat.com wrote: This is a proposal to add a new repository governed by the ironic inspector subteam. The current repository is named ahc-tools[1], however there is no attachment to this name. ironic-inspector-extra would seem to fit if this is moved under the Ironic umbrella. What is AHC? * AHC as a term comes from the enovance edeploy installation method[2]. * The general concept is
Re: [openstack-dev] [Ironic] Proposal to add a new repository
On 06/22/2015 04:19 PM, Devananda van der Veen wrote: Hi John, Thanks for the excellent summary! I found it very helpful to get caught up. I'd like to make sure I understand the direction ahc is going. A couple questions... Let me add my $0.5 :) I see that ahc is storing its information in swift. That's clever, but if Ironic provided a blob store for each node, would that be better? We discussed adding a search API to Ironic at the Vancouver summit, though no work has been done on that yet, afaik. If ahc is going to grow a REST API for searching for nodes based on specific criteria that it discovered, could/should we combine these within Ironic's API? I think John meant having API to replace scripts, so I guess search won't help. When we're talking about advanced matching, we're talking about the following: 1. We have a ramdisk tool (based on [8]) to get some insane of facts from withing the ramdisk (say, 1000 of them) 2. We have an Inspector plugin to put them all in Swift (or Ironic blob storage as above) 3. We have config files (aka rules) written in special JSON-alike DSL to do matching (one of the weak points is that these are files - I'd like API endpoint to accept these rules instead). 4. We have a script to run this DSL and get some output (match/not match + some matched variables - similar to what regexps do). As I understood it John want the latter to become an API endpoint, accepting rules (and maybe node UUIDs) and outputting some result. Not sure about benchmarking here, but again, it's probably an API endpoint that accepts some minimal expectations, and puts failed nodes to maintenance mode, if they fail to comply (again, that's how I understood it). It's not hard to make these API endpoints part of Inspector, but it's somewhat undesirable to have them optional... From a service coupling perspective, I like the approach that ahc is optional, and also that Ironic-inspector is optional, because this keeps the simple use-case for Ironic, well, simple! That said, this seems more like a configuration setting (should inspector do extra things?) than an entirely separate service, and separating them might be unnecessarily complicated. We keep thinking about it as well. After all, right now it's just a couple of utilities. There are 2 more concerns that initially made me pull out this code: 1. ahc-tools currently depends on the library [8], which I wish would be developed much more openly 2. it's cool that inspector is pluggable, but it has its cost: there's a poor feedback loop from inspector processing plugins to a user - like with all highly asynchronous code 3. it's also not possible (at least for now) to request a set of processing plugins when starting introspection via inspector. We solved the latter 2 problems by moving code to scripts. So now Inspector only puts some data to Swift, and scripts can do everything else. So now we've left with 1. dependency on hardware library 2. not very stable interface, much less stable than one of Inspector We still wonder how to solve these 2 without creating one more repository. Any ideas are welcome :) It sounds like this is the direction you'd like to go, and you took the current approach for expediency. If so, I'd like us to discuss a path to merge the functionality as it matures, and decide whether a separate repository is the right way to go long term. Thanks much, Devananda On Mon, Jun 22, 2015, 05:40 John Trowbridge tr...@redhat.com mailto:tr...@redhat.com wrote: This is a proposal to add a new repository governed by the ironic inspector subteam. The current repository is named ahc-tools[1], however there is no attachment to this name. ironic-inspector-extra would seem to fit if this is moved under the Ironic umbrella. What is AHC? * AHC as a term comes from the enovance edeploy installation method[2]. * The general concept is that we want to have a very granular picture of the physical hardware being used in a deployment in order to be able to match specific hardware to specific roles, as well as the ability to find poor performing outliers before we attempt to deploy. * For example: As a cloud operator, I want to make sure all logical disks have random read IOPs within 15% variance of each other. * The huge benefit of this tooling over current inspection is the number of facts collected (~1000 depending on the hardware), all of which can be used for matching. * Another example: As an end user, I would like to request a bare metal machine with a specific model GPU. What is ahc-tools? -- * We first tried to place all of this logic into a plugin in inspector[3] (discoverd at the time). [4] * This worked fine for just collecting some of the simple facts, however we now had a coupling between booting a ramdisk, and matching against the collected data. * ahc-tools
Re: [openstack-dev] [Ironic] Proposal to add a new repository
Hi, I see that ahc is storing its information in swift. That's clever, but if Ironic provided a blob store for each node, would that be better? I can try to answer that. The initial implementation was doing that but, AHC collect a fine-grained amount of data, e.g: * it runs benchmark on all disks using different IO size. Read being default but you can also test writing (disabled by default because it's destructive) * It tests memory performance with different blocks sizes (1K, 4K, 1M, 16M, 128M, 1G, 2G, ...) * It stress test each physical CPU (and all at once) So a machine with many disks, many CPUs and a lot of memory can generate a really big amount of data. For that reason storing the blob in Swift and just link the ID in Ironic is preferable. Cheers, Lucas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] Proposal to add a new repository
I'm On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge tr...@redhat.com wrote: On 06/22/2015 10:40 AM, Dmitry Tantsur wrote: On 06/22/2015 04:19 PM, Devananda van der Veen wrote: Hi John, Thanks for the excellent summary! I found it very helpful to get caught up. I'd like to make sure I understand the direction ahc is going. A couple questions... Thanks for your interest. Let me add my $0.5 :) I see that ahc is storing its information in swift. That's clever, but if Ironic provided a blob store for each node, would that be better? If the blob is large enough, this would be better. Originally we stored the data in the extra column of the Ironic db, but that proved disastrous: https://bugs.launchpad.net/ironic-inspector/+bug/1461252 We discussed adding a search API to Ironic at the Vancouver summit, though no work has been done on that yet, afaik. If ahc is going to grow a REST API for searching for nodes based on specific criteria that it discovered, could/should we combine these within Ironic's API? I think John meant having API to replace scripts, so I guess search won't help. When we're talking about advanced matching, we're talking about the following: 1. We have a ramdisk tool (based on [8]) to get some insane of facts from withing the ramdisk (say, 1000 of them) 2. We have an Inspector plugin to put them all in Swift (or Ironic blob storage as above) 3. We have config files (aka rules) written in special JSON-alike DSL to do matching (one of the weak points is that these are files - I'd like API endpoint to accept these rules instead). 4. We have a script to run this DSL and get some output (match/not match + some matched variables - similar to what regexps do). As I understood it John want the latter to become an API endpoint, accepting rules (and maybe node UUIDs) and outputting some result. Not sure about benchmarking here, but again, it's probably an API endpoint that accepts some minimal expectations, and puts failed nodes to maintenance mode, if they fail to comply (again, that's how I understood it). It's not hard to make these API endpoints part of Inspector, but it's somewhat undesirable to have them optional... From a service coupling perspective, I like the approach that ahc is optional, and also that Ironic-inspector is optional, because this keeps the simple use-case for Ironic, well, simple! That said, this seems more like a configuration setting (should inspector do extra things?) than an entirely separate service, and separating them might be unnecessarily complicated. We keep thinking about it as well. After all, right now it's just a couple of utilities. There are 2 more concerns that initially made me pull out this code: 1. ahc-tools currently depends on the library [8], which I wish would be developed much more openly 2. it's cool that inspector is pluggable, but it has its cost: there's a poor feedback loop from inspector processing plugins to a user - like with all highly asynchronous code 3. it's also not possible (at least for now) to request a set of processing plugins when starting introspection via inspector. We solved the latter 2 problems by moving code to scripts. So now Inspector only puts some data to Swift, and scripts can do everything else. So now we've left with 1. dependency on hardware library 2. not very stable interface, much less stable than one of Inspector We still wonder how to solve these 2 without creating one more repository. Any ideas are welcome :) Oh - good point. There's some neat looking functionality in enovance/hardware repository, but yea, until this is not a single-vendor-controlled repository, I think you made the right call. How would folks feel about moving that into openstack/ ? It is a goal of mine to solve issue 1 incrementally over time. Either by improving the library (both in function and in openness), or by slowly moving the implementation. That does not seem impossible to do within the inspector tree. Or that :) Either way, I agree with the direction -- moving the hardware inspection functionality into a common repository is good. If it makes sense to have two repositories (one for inspector, one for a hardware utils library) that's just fine with me. However, issue 2 is a fact. We currently have scripts, and we want to have a REST API. I do not see a transition between the two that does not involve a large amount of churn. From a quick read of the code, it looks like ahc-tools merely analyzes data already gathered by inspector enovance/hardware, providing some more advanced search/filtering/error-checking capabilities on the client side. If that read is correct, then I would like to align that work with my interest in developing a query-API for Ironic. It might take some time to do that, and so having a repo for client-side scripts is fine for now. (If that read is
Re: [openstack-dev] [Ironic] Proposal to add a new repository
Oh - one more thing. If ahc-tools depends on data gathered by enovance/hardware, then I'm not sure it makes sense to import one to openstack/ without the other. -Deva On Mon, Jun 22, 2015 at 5:08 PM Devananda van der Veen devananda@gmail.com wrote: I'm On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge tr...@redhat.com wrote: On 06/22/2015 10:40 AM, Dmitry Tantsur wrote: On 06/22/2015 04:19 PM, Devananda van der Veen wrote: Hi John, Thanks for the excellent summary! I found it very helpful to get caught up. I'd like to make sure I understand the direction ahc is going. A couple questions... Thanks for your interest. Let me add my $0.5 :) I see that ahc is storing its information in swift. That's clever, but if Ironic provided a blob store for each node, would that be better? If the blob is large enough, this would be better. Originally we stored the data in the extra column of the Ironic db, but that proved disastrous: https://bugs.launchpad.net/ironic-inspector/+bug/1461252 We discussed adding a search API to Ironic at the Vancouver summit, though no work has been done on that yet, afaik. If ahc is going to grow a REST API for searching for nodes based on specific criteria that it discovered, could/should we combine these within Ironic's API? I think John meant having API to replace scripts, so I guess search won't help. When we're talking about advanced matching, we're talking about the following: 1. We have a ramdisk tool (based on [8]) to get some insane of facts from withing the ramdisk (say, 1000 of them) 2. We have an Inspector plugin to put them all in Swift (or Ironic blob storage as above) 3. We have config files (aka rules) written in special JSON-alike DSL to do matching (one of the weak points is that these are files - I'd like API endpoint to accept these rules instead). 4. We have a script to run this DSL and get some output (match/not match + some matched variables - similar to what regexps do). As I understood it John want the latter to become an API endpoint, accepting rules (and maybe node UUIDs) and outputting some result. Not sure about benchmarking here, but again, it's probably an API endpoint that accepts some minimal expectations, and puts failed nodes to maintenance mode, if they fail to comply (again, that's how I understood it). It's not hard to make these API endpoints part of Inspector, but it's somewhat undesirable to have them optional... From a service coupling perspective, I like the approach that ahc is optional, and also that Ironic-inspector is optional, because this keeps the simple use-case for Ironic, well, simple! That said, this seems more like a configuration setting (should inspector do extra things?) than an entirely separate service, and separating them might be unnecessarily complicated. We keep thinking about it as well. After all, right now it's just a couple of utilities. There are 2 more concerns that initially made me pull out this code: 1. ahc-tools currently depends on the library [8], which I wish would be developed much more openly 2. it's cool that inspector is pluggable, but it has its cost: there's a poor feedback loop from inspector processing plugins to a user - like with all highly asynchronous code 3. it's also not possible (at least for now) to request a set of processing plugins when starting introspection via inspector. We solved the latter 2 problems by moving code to scripts. So now Inspector only puts some data to Swift, and scripts can do everything else. So now we've left with 1. dependency on hardware library 2. not very stable interface, much less stable than one of Inspector We still wonder how to solve these 2 without creating one more repository. Any ideas are welcome :) Oh - good point. There's some neat looking functionality in enovance/hardware repository, but yea, until this is not a single-vendor-controlled repository, I think you made the right call. How would folks feel about moving that into openstack/ ? It is a goal of mine to solve issue 1 incrementally over time. Either by improving the library (both in function and in openness), or by slowly moving the implementation. That does not seem impossible to do within the inspector tree. Or that :) Either way, I agree with the direction -- moving the hardware inspection functionality into a common repository is good. If it makes sense to have two repositories (one for inspector, one for a hardware utils library) that's just fine with me. However, issue 2 is a fact. We currently have scripts, and we want to have a REST API. I do not see a transition between the two that does not involve a large amount of churn. From a quick read of the code, it looks like ahc-tools merely analyzes data already gathered by inspector enovance/hardware, providing some more advanced
[openstack-dev] [Ironic] Proposal to add a new repository
This is a proposal to add a new repository governed by the ironic inspector subteam. The current repository is named ahc-tools[1], however there is no attachment to this name. ironic-inspector-extra would seem to fit if this is moved under the Ironic umbrella. What is AHC? * AHC as a term comes from the enovance edeploy installation method[2]. * The general concept is that we want to have a very granular picture of the physical hardware being used in a deployment in order to be able to match specific hardware to specific roles, as well as the ability to find poor performing outliers before we attempt to deploy. * For example: As a cloud operator, I want to make sure all logical disks have random read IOPs within 15% variance of each other. * The huge benefit of this tooling over current inspection is the number of facts collected (~1000 depending on the hardware), all of which can be used for matching. * Another example: As an end user, I would like to request a bare metal machine with a specific model GPU. What is ahc-tools? -- * We first tried to place all of this logic into a plugin in inspector[3] (discoverd at the time). [4] * This worked fine for just collecting some of the simple facts, however we now had a coupling between booting a ramdisk, and matching against the collected data. * ahc-tools started as a way to uncouple these two steps[5]. * We also added a wrapper around the enovance report tooling[6], as it already had the ability to generate reports based on the collected data, but was designed to read in the data from the filesystem. * The report tool has two functions. * First, it can group the systems by category (NICs, Firmware, Processors, etc.). * Second, it can use statistical analysis to find performance outliers. Why is ahc-tools useful to Ironic? -- * If we run benchmarks on hardware whenever it is turned back in by a tenant, we can easily put nodes into maintenance if the hardware is performing below some set threshold. This would allow us to have better certainty that the end user is getting what we promised them. * The advanced matching could also prove very useful. For VMs, I think the pets vs cattle analogy holds up very well, however many use cases for having cloud based bare metal involve access to specific hardware capabilities. I think advanced matching could help bridge this gap. Why not just put this code directly into inspector? --- * Clearly this code is 100% dependent on inspector. However, inspector is quite stable, and works great without any of this extra tooling. * ahc-tools is very immature, and will need many breaking changes to get to the same stability level of inspector. Why aren't you following the downstream-stackforge-openstack path? * This was the initial plan[7], however we were told that under the new big tent, that the openstack namespace is no longer meant to signify maturity of a project. * Instead, we were told we should propose the project directly to Ironic, or make a new separate project. What is the plan to make ahc-tools better? -- * The first major overhaul we would like to do is to put the reporting and matching functionality behind a REST API. * Reporting in particular will require significant work, as the current wrapper script wraps code that was never designed to be a library (Its output is just a series of print statements). One option is to improve the library[8] to be more library like, and the other is to reimplement the logic itself. Personally, while reimplementing the library is a large amount of work, I think it is probably worth the effort. * We would also like to add an API endpoint to coordinate distributed checks. For instance, if we want to confirm that there is physical network connectivity between a set of nodes, or if we would like to confirm the bandwidth of those connections. * The distributed checks and REST API will hopefully be completed in the Liberty timeframe. * Overhaul of the reporting will likely be an M target, unless there is interest from new contributors in working on this feature. * We are planning a talk for Tokyo on inspector that will also include details about this project. Thank you very much for your consideration. Respectfully, John Trowbridge [1] https://github.com/rdo-management/ahc-tools [2] https://github.com/enovance/edeploy/blob/master/docs/AHC.rst [3] https://github.com/openstack/ironic-inspector/commit/22a0e24efbef149377ea1e020f2d81968c10b58c [4] We can have out-of-tree plugins for the inspector, so some of this code might become a plugin again, but within the new repository tree. [5] https://github.com/openstack/ironic-inspector/commit/eaad7e09b99ab498e080e6e0ab71e69d00275422 [6] https://github.com/rdo-management/ahc-tools/blob/master/ahc_tools/report.py [7]
Re: [openstack-dev] [Ironic] Proposal to add a new repository
Hi John, Thanks for the excellent summary! I found it very helpful to get caught up. I'd like to make sure I understand the direction ahc is going. A couple questions... I see that ahc is storing its information in swift. That's clever, but if Ironic provided a blob store for each node, would that be better? We discussed adding a search API to Ironic at the Vancouver summit, though no work has been done on that yet, afaik. If ahc is going to grow a REST API for searching for nodes based on specific criteria that it discovered, could/should we combine these within Ironic's API? From a service coupling perspective, I like the approach that ahc is optional, and also that Ironic-inspector is optional, because this keeps the simple use-case for Ironic, well, simple! That said, this seems more like a configuration setting (should inspector do extra things?) than an entirely separate service, and separating them might be unnecessarily complicated. It sounds like this is the direction you'd like to go, and you took the current approach for expediency. If so, I'd like us to discuss a path to merge the functionality as it matures, and decide whether a separate repository is the right way to go long term. Thanks much, Devananda On Mon, Jun 22, 2015, 05:40 John Trowbridge tr...@redhat.com wrote: This is a proposal to add a new repository governed by the ironic inspector subteam. The current repository is named ahc-tools[1], however there is no attachment to this name. ironic-inspector-extra would seem to fit if this is moved under the Ironic umbrella. What is AHC? * AHC as a term comes from the enovance edeploy installation method[2]. * The general concept is that we want to have a very granular picture of the physical hardware being used in a deployment in order to be able to match specific hardware to specific roles, as well as the ability to find poor performing outliers before we attempt to deploy. * For example: As a cloud operator, I want to make sure all logical disks have random read IOPs within 15% variance of each other. * The huge benefit of this tooling over current inspection is the number of facts collected (~1000 depending on the hardware), all of which can be used for matching. * Another example: As an end user, I would like to request a bare metal machine with a specific model GPU. What is ahc-tools? -- * We first tried to place all of this logic into a plugin in inspector[3] (discoverd at the time). [4] * This worked fine for just collecting some of the simple facts, however we now had a coupling between booting a ramdisk, and matching against the collected data. * ahc-tools started as a way to uncouple these two steps[5]. * We also added a wrapper around the enovance report tooling[6], as it already had the ability to generate reports based on the collected data, but was designed to read in the data from the filesystem. * The report tool has two functions. * First, it can group the systems by category (NICs, Firmware, Processors, etc.). * Second, it can use statistical analysis to find performance outliers. Why is ahc-tools useful to Ironic? -- * If we run benchmarks on hardware whenever it is turned back in by a tenant, we can easily put nodes into maintenance if the hardware is performing below some set threshold. This would allow us to have better certainty that the end user is getting what we promised them. * The advanced matching could also prove very useful. For VMs, I think the pets vs cattle analogy holds up very well, however many use cases for having cloud based bare metal involve access to specific hardware capabilities. I think advanced matching could help bridge this gap. Why not just put this code directly into inspector? --- * Clearly this code is 100% dependent on inspector. However, inspector is quite stable, and works great without any of this extra tooling. * ahc-tools is very immature, and will need many breaking changes to get to the same stability level of inspector. Why aren't you following the downstream-stackforge-openstack path? * This was the initial plan[7], however we were told that under the new big tent, that the openstack namespace is no longer meant to signify maturity of a project. * Instead, we were told we should propose the project directly to Ironic, or make a new separate project. What is the plan to make ahc-tools better? -- * The first major overhaul we would like to do is to put the reporting and matching functionality behind a REST API. * Reporting in particular will require significant work, as the current wrapper script wraps code that was never designed to be a library (Its output is just a series of print statements). One option is to improve the library[8] to
