Thanks, Gavin. You have seconds. I'll contact you offline to
get you set up.
On Thu, 10 May 2007, Gavin Maltby wrote:
The Project
This project proposes improving the generic support of x86
CPUs that implement the Machine Check Architecture (MCA). It
also proposes to create a model-specific plugin interface whereby
model-specific MCA implementation may supplement and extend
the generic support rather than stand alone as today. Finally, the
project proposes to redesign the cpu-module interface such that
the MCA support it provides will be usable in both
virtualized and bare-metal environments.
Background
Prior to the "amd64 FMA" project (putback in February 2006)
support for MCA was woeful, with no intelligent enabling of
error detectors, no polling for errors that do not
produce a machine check, and a clumsy machine-check handler
which barfed an incomplete and raw dump of error telemetry to
the console before panic. There was no model-specific MCA
support whatsoever.
The "amd64 FMA" project, putback in February 2006, delivered
detailed model-specific support for AMD family 0xf revisions B to E
(also known as AMD Opteron/Athlon64/Turion64, K8 etc; revisions
B to E all have DDR1 memory) and moderately improved generic
MCA support.
The AMD family family 0xf revs B to E support included:
- performing MCA and model-specific configuration
beyond what the BIOS had setup
- enabling all MCA banks and machine-check on all
uncorrectable error types
- an error poller for errors that do not machine-check
(such as correctable memory errors)
- an error handler that collects all telemetry and
bakes it into a model-specific error report which
is dispatched for logging and diagnosis via a robust
mechanism that will carry telemetry across a panic/reboot
cycle
- detailed knowledge of model-specific error types, able
to produce an ereport error class from the raw telemetry
and to specify the appropriate ereport payload members
to include for that error class
- a memory-controller driver able to discover all aspects
of memory configuration and to map system addresses to
memory resource
- a topology enumerator to describe cpu and memory topology
and properties to the topology library, for use in
diagnosis software
- eversholt diagnosis rules to diagnose all telemetry
- corresponding fault classes and sun.com/msg knowledge
articles
The generic support, applied to anything other than the AMD models
above including all Intel processors and any AMD models outside the
model range above, included:
- generic MCA detector and error type enabling
- a machine check exception handler which dumps all
MCA telemetry to the console, still in raw form
- no error poller
- no preparation of ereports (even for panic events), and
therefore no diagnosis of any kind
The above support was backported to Solaris 10 Update 2.
The model-specific AMD support was updated in build 51 to include
AMD family 0xf revision F and G support. Those revisions use
DDR2 memory, and most work was required in the memory-controller
driver. This additonal support has been backported to Solaris 10
Update 4 (pending release at this time).
The Problem
1) The first problem is that the generic cpu MCA support is light
on features:
- it lacks an error poller; we won't log memory
correctable ECC errors, for example
- it does not raise ereports for logging and diagnosis,
even for errors that result in a panic
- on an error that results in a panic at best you have
some raw, uninterpreted error register values and no
error classification or interpretation; if the system
lacks console logging no evidence at all may remain
when the system resets.
This generic support is applied to all Intel processors, and to
AMD processors not handled by already-delivered AMD model-specific
support. Where a platform with full model-specific support
is upgradeable to new processors that drop in to existing sockets
and where the new processor family/model/stepping is no longer
covered by the existing model-specific support, the platform
after upgrade will regress to the dumb generic cpu support.
So for all Intel and for new AMD processors there is a window of
OpenSolaris builds during which MCA support is very light - it
takes some time to implement full model-specific support for a cpu.
Improving generic cpu support will raise the base level of support
for all processors that implement MCA/MCE features. Purely from the
Sun point of view, and not a concern for OpenSolaris itself, the
delay in introducing new model-specific support is further
stretched in having to backport to a suitable Solaris 10 Update
release; it is also possible that products based on a new
cpu model can revenue-release before corresponding full FMA
support for the cpu model is available in Solaris 10.
2) The second problem is that the current cpu module
implementations (generic and AMD family 0xf) and the
cpu module interface itself are completely invalildated when
not running natively. A separate project will be proposed for
FMA/Xen support, but generic cpu MCA support and the cpu module
interface changes required for the current project proposal
should be made with that requirement in mind.
3) A third problem is that the current cpu module design
leads to substantial code duplication across model-specific
implementations (e.g., implementing an error poller in each).
This should not be necessary given the common base MCA that they
adhere to.
Key Objectives
1) Enrich the generic cpu support to include an error poller,
error type classification (as far as the architectural definition
allows), raising of ereports capturing all the telemetry for
logging and some degree of diagnosis (again as much as is possible
generically).
2) Design a new cpu model-specific MCA module interface which
will supplement generic cpu MCA support to avoid code duplication
and improve maintainability. This new interface should be
rich enough to allow the refactoring and redelivery of exising
AMD model-specific support, and should be designed with upcoming
Intel model-specific support in mind.
3) Identify vendor-specific but nevertheless fairly widely-
applicable (to many models from that vendor) features that
can be captured in an intermediate level of model-specific
support for new CPU models from that vendor. For example in
the absence of a fully-featured memory-controller driver we may
be unable to determine the full memory topology and provide
full error address to memory resource translation, but both
AMD and Intel in recent models have hardware registers that
can identify where errors have occured (albeit at a lower
resolution than a full memory-controller driver may be able to
provide). Such support could provide a level of "generic AMD"
and "generic Intel" support for new cpu releases that is
better than the lowest common denominator generic support,
and which is later not loaded in favour of full model-specific
support once that is available.
4) Redesign the cpu module interface to cope with running both
natively and in a virtualized setting, Xen Solaris domain 0
in particular. The new interface should not assume any 1:1
relationship between Solaris logical cpus (cpu_t structures)
and real physical cores on the system, should not assume that
being bound to a Solaris cpu means we can perform reads from
hardware registers all from the same real core, etc.
Instead the points at which we acquire telemetry and other
hardware details should be well-defined, and the information
gathered via calls into a newly-defined native or virtualized layer.
Implementation of the virtualized layer itself is left to another
project.
5) The project deliverables must include some degree of hardware
error-injection, or of error simulation if that is not possible.
Existing test harnesses should be extended to include generic
cpu scenarios.
6) As with any FMA project, this should include an FMA portfolio
detailing all events, diagnosis thereof, ASRU boundaries etc.
To date there are no open FMA portfolios so those outside of Sun
will have little idea of what this involves - that will become
clear during the progres of the project.
7) This project will not deliver full model-specific support
for any Intel cpus nor for upcoming "AMD family 0x10" processors.
It should, however, proceed with the needs of those projects
in mind and should colloborate with those projects to understand
those needs.
Community Participation
Participation and contribution from the OpenSolaris community is
encouraged. This project will use public documentation from both
AMD and Intel, available from the websites of those companies.
As such, there is no vendor NDA or similar barrier to community
participation. It may be necessary for some design decisions to
be made by or in conjunction with those with access to NDA
documentation, but these are expected to be the exception.
Error injector sources are currently in the closed tree. Otherwise
all applicable source is available in OpenSolaris.
The event registry is now available externally to Sun, so there is no
reason that the FMA portfolio cannot be prepared and published
externally as part of this project.
Gavin
--
Gavin Maltby, Solaris Kernel Development.
_______________________________________________
opensolaris-discuss mailing list
[email protected]
_______________________________________________
opensolaris-discuss mailing list
[email protected]
