I'm On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge <[email protected]> wrote:
> > > On 06/22/2015 10:40 AM, Dmitry Tantsur wrote: > > On 06/22/2015 04:19 PM, Devananda van der Veen wrote: > >> Hi John, > >> > >> Thanks for the excellent summary! I found it very helpful to get caught > >> up. I'd like to make sure I understand the direction ahc is going. A > >> couple questions... > > Thanks for your interest. > > > > > Let me add my $0.5 :) > > > >> > >> I see that ahc is storing its information in swift. That's clever, but > >> if Ironic provided a blob store for each node, would that be better? > > If the blob is large enough, this would be better. Originally we stored > the data in the extra column of the Ironic db, but that proved disastrous: > > https://bugs.launchpad.net/ironic-inspector/+bug/1461252 > > >> > >> We discussed adding a search API to Ironic at the Vancouver summit, > >> though no work has been done on that yet, afaik. If ahc is going to grow > >> a REST API for searching for nodes based on specific criteria that it > >> discovered, could/should we combine these within Ironic's API? > > > > I think John meant having API to replace scripts, so I guess search > > won't help. When we're talking about advanced matching, we're talking > > about the following: > > 1. We have a ramdisk tool (based on [8]) to get some insane of facts > > from withing the ramdisk (say, 1000 of them) > > 2. We have an Inspector plugin to put them all in Swift (or Ironic blob > > storage as above) > > 3. We have config files (aka rules) written in special JSON-alike DSL to > > do matching (one of the weak points is that these are files - I'd like > > API endpoint to accept these rules instead). > > 4. We have a script to run this DSL and get some output (match/not match > > + some matched variables - similar to what regexps do). > > As I understood it John want the latter to become an API endpoint, > > accepting rules (and maybe node UUIDs) and outputting some result. > > > > Not sure about benchmarking here, but again, it's probably an API > > endpoint that accepts some minimal expectations, and puts failed nodes > > to maintenance mode, if they fail to comply (again, that's how I > > understood it). > > > > It's not hard to make these API endpoints part of Inspector, but it's > > somewhat undesirable to have them optional... > > > >> > >> From a service coupling perspective, I like the approach that ahc is > >> optional, and also that Ironic-inspector is optional, because this keeps > >> the simple use-case for Ironic, well, simple! That said, this seems more > >> like a configuration setting (should inspector do extra things?) than an > >> entirely separate service, and separating them might be unnecessarily > >> complicated. > > > > We keep thinking about it as well. After all, right now it's just a > > couple of utilities. There are 2 more concerns that initially made me > > pull out this code: > > 1. ahc-tools currently depends on the library [8], which I wish would be > > developed much more openly > > 2. it's cool that inspector is pluggable, but it has its cost: there's a > > poor feedback loop from inspector processing plugins to a user - like > > with all highly asynchronous code > > 3. it's also not possible (at least for now) to request a set of > > processing plugins when starting introspection via inspector. > > > > We solved the latter 2 problems by moving code to scripts. So now > > Inspector only puts some data to Swift, and scripts can do everything > else. > > > > So now we've left with > > 1. dependency on "hardware" library > > 2. not very stable interface, much less stable than one of Inspector > > > > We still wonder how to solve these 2 without creating one more > > repository. Any ideas are welcome :) > > Oh - good point. There's some neat looking functionality in enovance/hardware repository, but yea, until this is not a single-vendor-controlled repository, I think you made the right call. How would folks feel about moving that into openstack/ ? > It is a goal of mine to solve issue 1 incrementally over time. Either by > improving the library (both in function and in openness), or by slowly > moving the implementation. That does not seem impossible to do within > the inspector tree. > Or that :) Either way, I agree with the direction -- moving the hardware inspection functionality into a common repository is good. If it makes sense to have two repositories (one for inspector, one for a hardware utils library) that's just fine with me. However, issue 2 is a fact. We currently have scripts, and we want to > have a REST API. I do not see a transition between the two that does not > involve a large amount of churn. > >From a quick read of the code, it looks like ahc-tools merely analyzes data already gathered by inspector & enovance/hardware, providing some more advanced search/filtering/error-checking capabilities on the client side. If that read is correct, then I would like to align that work with my interest in developing a query-API for Ironic. It might take some time to do that, and so having a repo for client-side scripts is fine for now. (If that read is incorrect, please explain what I missed) > > I am not sure how to solve issue 2 without a separate repository. I do > think there is a logical separation of concerns though, so we may not > need to completely merge the two in the future. Inspector collects data, > and ahc-tools (or whatever it is eventually named) is used to act on the > data. > Right -- I see inspector as a separate process from Ironic, but it sounds like ahc-tools could eventually be functionality provided by Ironic itself, eg, GET /v1/search/ with a body containing some query document. Or something.... -Deva > > >> > >> It sounds like this is the direction you'd like to go, and you took the > >> current approach for expediency. If so, I'd like us to discuss a path to > >> merge the functionality as it matures, and decide whether a separate > >> repository is the right way to go long term. > >> > >> Thanks much, > >> Devananda > >> > >> > >> On Mon, Jun 22, 2015, 05:40 John Trowbridge <[email protected] > >> <mailto:[email protected]>> wrote: > >> > >> This is a proposal to add a new repository governed by the ironic > >> inspector subteam. The current repository is named ahc-tools[1], > >> however > >> there is no attachment to this name. "ironic-inspector-extra" > >> would seem > >> to fit if this is moved under the Ironic umbrella. > >> > >> What is AHC? > >> ------------ > >> * AHC as a term comes from the enovance edeploy installation > >> method[2]. > >> * The general concept is that we want to have a very granular > >> picture of > >> the physical hardware being used in a deployment in order to be > >> able to > >> match specific hardware to specific roles, as well as the ability to > >> find poor performing outliers before we attempt to deploy. > >> * For example: As a cloud operator, I want to make sure all logical > >> disks have random read IOPs within 15% variance of each other. > >> * The huge benefit of this tooling over current inspection is the > >> number > >> of facts collected (~1000 depending on the hardware), all of which > >> can > >> be used for matching. > >> * Another example: As an end user, I would like to request a bare > >> metal > >> machine with a specific model GPU. > >> > >> What is ahc-tools? > >> ------------------ > >> * We first tried to place all of this logic into a plugin in > >> inspector[3] (discoverd at the time). [4] > >> * This worked fine for just collecting some of the simple facts, > >> however > >> we now had a coupling between booting a ramdisk, and matching > against > >> the collected data. > >> * ahc-tools started as a way to uncouple these two steps[5]. > >> * We also added a wrapper around the enovance report tooling[6], > >> as it > >> already had the ability to generate reports based on the collected > >> data, > >> but was designed to read in the data from the filesystem. > >> * The report tool has two functions. > >> * First, it can group the systems by category (NICs, Firmware, > >> Processors, etc.). > >> * Second, it can use statistical analysis to find performance > >> outliers. > >> > >> Why is ahc-tools useful to Ironic? > >> ---------------------------------- > >> * If we run benchmarks on hardware whenever it is turned back in by > a > >> tenant, we can easily put nodes into maintenance if the hardware is > >> performing below some set threshold. This would allow us to have > >> better > >> certainty that the end user is getting what we promised them. > >> * The advanced matching could also prove very useful. For VMs, I > >> think > >> the pets vs cattle analogy holds up very well, however many use > cases > >> for having cloud based bare metal involve access to specific > hardware > >> capabilities. I think advanced matching could help bridge this gap. > >> > >> Why not just put this code directly into inspector? > >> --------------------------------------------------- > >> * Clearly this code is 100% dependent on inspector. However, > >> inspector > >> is quite stable, and works great without any of this extra tooling. > >> * ahc-tools is very immature, and will need many breaking changes > >> to get > >> to the same stability level of inspector. > >> > >> Why aren't you following the downstream->stackforge->openstack path? > >> -------------------------------------------------------------------- > >> * This was the initial plan[7], however we were told that under > >> the new > >> "big tent", that the openstack namespace is no longer meant to > >> signify > >> maturity of a project. > >> * Instead, we were told we should propose the project directly to > >> Ironic, or make a new separate project. > >> > >> What is the plan to make ahc-tools better? > >> ------------------------------------------ > >> * The first major overhaul we would like to do is to put the > >> reporting > >> and matching functionality behind a REST API. > >> * Reporting in particular will require significant work, as the > >> current > >> wrapper script wraps code that was never designed to be a library > >> (Its > >> output is just a series of print statements). One option is to > >> improve > >> the library[8] to be more library like, and the other is to > >> reimplement > >> the logic itself. Personally, while reimplementing the library is a > >> large amount of work, I think it is probably worth the effort. > >> * We would also like to add an API endpoint to coordinate > distributed > >> checks. For instance, if we want to confirm that there is physical > >> network connectivity between a set of nodes, or if we would like to > >> confirm the bandwidth of those connections. > >> * The distributed checks and REST API will hopefully be completed > >> in the > >> Liberty timeframe. > >> * Overhaul of the reporting will likely be an M target, unless > >> there is > >> interest from new contributors in working on this feature. > >> * We are planning a talk for Tokyo on inspector that will also > >> include > >> details about this project. > >> > >> Thank you very much for your consideration. > >> > >> Respectfully, > >> John Trowbridge > >> > >> [1] https://github.com/rdo-management/ahc-tools > >> [2] https://github.com/enovance/edeploy/blob/master/docs/AHC.rst > >> [3] > >> > >> > https://github.com/openstack/ironic-inspector/commit/22a0e24efbef149377ea1e020f2d81968c10b58c > >> > >> [4] We can have out-of-tree plugins for the inspector, so some of > >> this > >> code might become a plugin again, but within the new repository > tree. > >> [5] > >> > >> > https://github.com/openstack/ironic-inspector/commit/eaad7e09b99ab498e080e6e0ab71e69d00275422 > >> > >> [6] > >> > >> > https://github.com/rdo-management/ahc-tools/blob/master/ahc_tools/report.py > >> > >> [7] https://review.openstack.org/#/c/193392/ > >> [8] https://github.com/enovance/hardware > >> > >> > >> > __________________________________________________________________________ > >> > >> OpenStack Development Mailing List (not for usage questions) > >> Unsubscribe: > >> [email protected]?subject:unsubscribe > >> > >> <http://[email protected]?subject:unsubscribe> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >> > >> > >> > __________________________________________________________________________ > >> > >> OpenStack Development Mailing List (not for usage questions) > >> Unsubscribe: > >> [email protected]?subject:unsubscribe > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > [email protected]?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
