On 6/25/19 11:55 AM, Corey Minyard wrote:
On Tue, Jun 25, 2019 at 10:24:29AM -0400, Tony Camuso wrote:
I'd like to know if there are any test suites out there for
automated IPMI testing, especially as new features emerge and
new bugs are exposed.
We have a few of our own, but they are relatively primitive
and don't provide a breadth and depth of coverage that could
affirm full functionality or catch all regressions.
Unfortunately, many regressions and functionality misses are
caught by our customers.
I know that Corey uses a VM with virtual IPMI, and I'd like
to know more about that, if there is a link somewhere that
has the information.
qemu has an IPMI simulator built into it, either with a local minimal
BMC (basically with just a watchdog) and it allows connecting to an
external BMC (like ipmi_sim in openipmi) for more complex simulation.
In fact, some people are using this to do cluster management. If you
search "qemu openipmi" you will find a number of resources.
I also have a lot of pending extensions to the qemu IPMI simulator,
for SSIF and PCI interfaces, for instance, that I haven't found time
to get ready.
ipmi_sim also provides LAN connections, so it allows testing LAN interfaces
with simulation.
Thanks for the great leads on qemu and ipmi_sim. I'll be looking into
that directly.
Virtual technology frees us from having to find systems using
all the various methods of reporting the IPMI as well as a
means to test all the functionality, and it provides a better
way to automate testing.
Indeed it has been very helpful for me. That's why I wrote it :-).
But I'm not sure what you mean by testing here, many components need
to be tested:
* BMC
* Driver
* Libraries
* Applications
That's a good question. Our focus in the kernel group would be on the
driver. However, developing testing for the hardware, libraries, and
apps would be helpful.
Testing BMCs is probably the weakest link. From my experience, most vendors>
just get IPMI working to so that their own management applications will work
and don't properly implement all the things that are required for a general
application to work (entity presence and entity containment are two that are
notoriously bad).
+1
A suite of tests for the hardware might help make vendors more sensitive to
spec compliance.
I know Intel has some sort of test suite, but I don't
know much about it, and you really can't test all that stuff easily, though
I suppose you could test a lot. It would be hard.
A driver test suite would be really nice, and that something that is
probably fairly doable. I don't envision the driver changing very much
in the future, though. It's really something I should have done long
ago :(. The difficulty here is that many issues I see are creative
interpretations of the standard in implementations.
I'd like to attempt a driver test suite. Any suggestions or contributions
are welcome.
The OpenIPMI library has some built-in tests (make check) using ipmi_sim.
The exercise basic functionality in most places, but aren't complete by
any means. I don't know if ipmitool or freeipmi have anything.
It would be really nice if applications could have a way to use a simulator
to test, and ipmi_sim could easily provide that (either through qemu or
through a LAN interface). It's not very easy to use, though, it needs a
lot of work and documentation to make it user-friendly. Setting up a BMC
with all the sensors, SDRs, FRU data, etc. is a big job, even with a
user-friendly tool.
I'm happy to work with volunteers on any of these things. I don't have a
lot of time to spend on anything like this in the near future. IPMI doesn't
seem to be going away.
-corey
Many thanks for the comprehensive reply and suggestions.
Regards,
Tony Camuso
Red Hat
_______________________________________________
Openipmi-developer mailing list
Openipmi-developer@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/openipmi-developer
