Oh - one more thing. If ahc-tools depends on data gathered by enovance/hardware, then I'm not sure it makes sense to import one to openstack/ without the other.
-Deva On Mon, Jun 22, 2015 at 5:08 PM Devananda van der Veen < [email protected]> wrote: > I'm > On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge <[email protected]> wrote: > >> >> >> On 06/22/2015 10:40 AM, Dmitry Tantsur wrote: >> > On 06/22/2015 04:19 PM, Devananda van der Veen wrote: >> >> Hi John, >> >> >> >> Thanks for the excellent summary! I found it very helpful to get caught >> >> up. I'd like to make sure I understand the direction ahc is going. A >> >> couple questions... >> >> Thanks for your interest. >> >> > >> > Let me add my $0.5 :) >> > >> >> >> >> I see that ahc is storing its information in swift. That's clever, but >> >> if Ironic provided a blob store for each node, would that be better? >> >> If the blob is large enough, this would be better. Originally we stored >> the data in the extra column of the Ironic db, but that proved disastrous: >> >> https://bugs.launchpad.net/ironic-inspector/+bug/1461252 >> > > >> >> >> >> We discussed adding a search API to Ironic at the Vancouver summit, >> >> though no work has been done on that yet, afaik. If ahc is going to >> grow >> >> a REST API for searching for nodes based on specific criteria that it >> >> discovered, could/should we combine these within Ironic's API? >> > >> > I think John meant having API to replace scripts, so I guess search >> > won't help. When we're talking about advanced matching, we're talking >> > about the following: >> > 1. We have a ramdisk tool (based on [8]) to get some insane of facts >> > from withing the ramdisk (say, 1000 of them) >> > 2. We have an Inspector plugin to put them all in Swift (or Ironic blob >> > storage as above) >> > 3. We have config files (aka rules) written in special JSON-alike DSL to >> > do matching (one of the weak points is that these are files - I'd like >> > API endpoint to accept these rules instead). >> > 4. We have a script to run this DSL and get some output (match/not match >> > + some matched variables - similar to what regexps do). >> > As I understood it John want the latter to become an API endpoint, >> > accepting rules (and maybe node UUIDs) and outputting some result. >> > >> > Not sure about benchmarking here, but again, it's probably an API >> > endpoint that accepts some minimal expectations, and puts failed nodes >> > to maintenance mode, if they fail to comply (again, that's how I >> > understood it). >> > >> > It's not hard to make these API endpoints part of Inspector, but it's >> > somewhat undesirable to have them optional... >> > >> >> >> >> From a service coupling perspective, I like the approach that ahc is >> >> optional, and also that Ironic-inspector is optional, because this >> keeps >> >> the simple use-case for Ironic, well, simple! That said, this seems >> more >> >> like a configuration setting (should inspector do extra things?) than >> an >> >> entirely separate service, and separating them might be unnecessarily >> >> complicated. >> > >> > We keep thinking about it as well. After all, right now it's just a >> > couple of utilities. There are 2 more concerns that initially made me >> > pull out this code: >> > 1. ahc-tools currently depends on the library [8], which I wish would be >> > developed much more openly >> > > > 2. it's cool that inspector is pluggable, but it has its cost: there's a >> > poor feedback loop from inspector processing plugins to a user - like >> > with all highly asynchronous code >> > 3. it's also not possible (at least for now) to request a set of >> > processing plugins when starting introspection via inspector. >> > >> > We solved the latter 2 problems by moving code to scripts. So now >> > Inspector only puts some data to Swift, and scripts can do everything >> else. >> > >> > So now we've left with >> > 1. dependency on "hardware" library >> > 2. not very stable interface, much less stable than one of Inspector >> > >> > We still wonder how to solve these 2 without creating one more >> > repository. Any ideas are welcome :) >> >> > Oh - good point. There's some neat looking functionality in > enovance/hardware repository, but yea, until this is not a > single-vendor-controlled repository, I think you made the right call. > > How would folks feel about moving that into openstack/ ? > > >> It is a goal of mine to solve issue 1 incrementally over time. Either by >> improving the library (both in function and in openness), or by slowly >> moving the implementation. That does not seem impossible to do within >> the inspector tree. >> > > Or that :) Either way, I agree with the direction -- moving the hardware > inspection functionality into a common repository is good. If it makes > sense to have two repositories (one for inspector, one for a hardware utils > library) that's just fine with me. > > > However, issue 2 is a fact. We currently have scripts, and we want to >> have a REST API. I do not see a transition between the two that does not >> involve a large amount of churn. >> > > From a quick read of the code, it looks like ahc-tools merely analyzes > data already gathered by inspector & enovance/hardware, providing some more > advanced search/filtering/error-checking capabilities on the client side. > If that read is correct, then I would like to align that work with my > interest in developing a query-API for Ironic. It might take some time to > do that, and so having a repo for client-side scripts is fine for now. > > (If that read is incorrect, please explain what I missed) > > >> >> I am not sure how to solve issue 2 without a separate repository. I do >> think there is a logical separation of concerns though, so we may not >> need to completely merge the two in the future. Inspector collects data, >> and ahc-tools (or whatever it is eventually named) is used to act on the >> data. >> > > Right -- I see inspector as a separate process from Ironic, but it sounds > like ahc-tools could eventually be functionality provided by Ironic itself, > eg, GET /v1/search/ with a body containing some query document. Or > something.... > > -Deva > > > > >> >> >> >> It sounds like this is the direction you'd like to go, and you took the >> >> current approach for expediency. If so, I'd like us to discuss a path >> to >> >> merge the functionality as it matures, and decide whether a separate >> >> repository is the right way to go long term. >> >> >> >> Thanks much, >> >> Devananda >> >> >> >> >> >> On Mon, Jun 22, 2015, 05:40 John Trowbridge <[email protected] >> >> <mailto:[email protected]>> wrote: >> >> >> >> This is a proposal to add a new repository governed by the ironic >> >> inspector subteam. The current repository is named ahc-tools[1], >> >> however >> >> there is no attachment to this name. "ironic-inspector-extra" >> >> would seem >> >> to fit if this is moved under the Ironic umbrella. >> >> >> >> What is AHC? >> >> ------------ >> >> * AHC as a term comes from the enovance edeploy installation >> >> method[2]. >> >> * The general concept is that we want to have a very granular >> >> picture of >> >> the physical hardware being used in a deployment in order to be >> >> able to >> >> match specific hardware to specific roles, as well as the ability >> to >> >> find poor performing outliers before we attempt to deploy. >> >> * For example: As a cloud operator, I want to make sure all logical >> >> disks have random read IOPs within 15% variance of each other. >> >> * The huge benefit of this tooling over current inspection is the >> >> number >> >> of facts collected (~1000 depending on the hardware), all of which >> >> can >> >> be used for matching. >> >> * Another example: As an end user, I would like to request a bare >> >> metal >> >> machine with a specific model GPU. >> >> >> >> What is ahc-tools? >> >> ------------------ >> >> * We first tried to place all of this logic into a plugin in >> >> inspector[3] (discoverd at the time). [4] >> >> * This worked fine for just collecting some of the simple facts, >> >> however >> >> we now had a coupling between booting a ramdisk, and matching >> against >> >> the collected data. >> >> * ahc-tools started as a way to uncouple these two steps[5]. >> >> * We also added a wrapper around the enovance report tooling[6], >> >> as it >> >> already had the ability to generate reports based on the collected >> >> data, >> >> but was designed to read in the data from the filesystem. >> >> * The report tool has two functions. >> >> * First, it can group the systems by category (NICs, Firmware, >> >> Processors, etc.). >> >> * Second, it can use statistical analysis to find performance >> >> outliers. >> >> >> >> Why is ahc-tools useful to Ironic? >> >> ---------------------------------- >> >> * If we run benchmarks on hardware whenever it is turned back in >> by a >> >> tenant, we can easily put nodes into maintenance if the hardware is >> >> performing below some set threshold. This would allow us to have >> >> better >> >> certainty that the end user is getting what we promised them. >> >> * The advanced matching could also prove very useful. For VMs, I >> >> think >> >> the pets vs cattle analogy holds up very well, however many use >> cases >> >> for having cloud based bare metal involve access to specific >> hardware >> >> capabilities. I think advanced matching could help bridge this gap. >> >> >> >> Why not just put this code directly into inspector? >> >> --------------------------------------------------- >> >> * Clearly this code is 100% dependent on inspector. However, >> >> inspector >> >> is quite stable, and works great without any of this extra tooling. >> >> * ahc-tools is very immature, and will need many breaking changes >> >> to get >> >> to the same stability level of inspector. >> >> >> >> Why aren't you following the downstream->stackforge->openstack >> path? >> >> >> -------------------------------------------------------------------- >> >> * This was the initial plan[7], however we were told that under >> >> the new >> >> "big tent", that the openstack namespace is no longer meant to >> >> signify >> >> maturity of a project. >> >> * Instead, we were told we should propose the project directly to >> >> Ironic, or make a new separate project. >> >> >> >> What is the plan to make ahc-tools better? >> >> ------------------------------------------ >> >> * The first major overhaul we would like to do is to put the >> >> reporting >> >> and matching functionality behind a REST API. >> >> * Reporting in particular will require significant work, as the >> >> current >> >> wrapper script wraps code that was never designed to be a library >> >> (Its >> >> output is just a series of print statements). One option is to >> >> improve >> >> the library[8] to be more library like, and the other is to >> >> reimplement >> >> the logic itself. Personally, while reimplementing the library is a >> >> large amount of work, I think it is probably worth the effort. >> >> * We would also like to add an API endpoint to coordinate >> distributed >> >> checks. For instance, if we want to confirm that there is physical >> >> network connectivity between a set of nodes, or if we would like to >> >> confirm the bandwidth of those connections. >> >> * The distributed checks and REST API will hopefully be completed >> >> in the >> >> Liberty timeframe. >> >> * Overhaul of the reporting will likely be an M target, unless >> >> there is >> >> interest from new contributors in working on this feature. >> >> * We are planning a talk for Tokyo on inspector that will also >> >> include >> >> details about this project. >> >> >> >> Thank you very much for your consideration. >> >> >> >> Respectfully, >> >> John Trowbridge >> >> >> >> [1] https://github.com/rdo-management/ahc-tools >> >> [2] https://github.com/enovance/edeploy/blob/master/docs/AHC.rst >> >> [3] >> >> >> >> >> https://github.com/openstack/ironic-inspector/commit/22a0e24efbef149377ea1e020f2d81968c10b58c >> >> >> >> [4] We can have out-of-tree plugins for the inspector, so some of >> >> this >> >> code might become a plugin again, but within the new repository >> tree. >> >> [5] >> >> >> >> >> https://github.com/openstack/ironic-inspector/commit/eaad7e09b99ab498e080e6e0ab71e69d00275422 >> >> >> >> [6] >> >> >> >> >> https://github.com/rdo-management/ahc-tools/blob/master/ahc_tools/report.py >> >> >> >> [7] https://review.openstack.org/#/c/193392/ >> >> [8] https://github.com/enovance/hardware >> >> >> >> >> >> >> __________________________________________________________________________ >> >> >> >> OpenStack Development Mailing List (not for usage questions) >> >> Unsubscribe: >> >> [email protected]?subject:unsubscribe >> >> >> >> <http://[email protected]?subject:unsubscribe> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> >> >> >> >> >> __________________________________________________________________________ >> >> >> >> OpenStack Development Mailing List (not for usage questions) >> >> Unsubscribe: >> >> [email protected]?subject:unsubscribe >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> > >> > >> > >> __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> [email protected]?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> [email protected]?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
