I'm sponsoring the following fast-track for Eric Schrock and Keith
Wesolowski to introduce new Private libscsi and libses libraries.
The case timer is set for 3/19 and the binding is for a Patch release.
A copy of the spec (below) is also saved in the incept.materials
directory, and I've also put copies of header files there.
-Mike
---8<---
A. DESCRIPTION
On Solaris, userland applications wishing to interact with SES devices currently
use a combination of uscsi(7I) and ses(7D). Both of these interfaces
are difficult to use, and require significant infrastructure in each
application. As part of the Sun storage strategy, encompassing both work in
Solaris and in the Systems group, it is necessary to create libraries that
hide such implementation details and provide clean interfaces on which higher
level applications can be developed. The initial consumers of this will
be the SES libtopo FMA enumerator and the unbundled CAM storage management
utility, but the libraries are being delivered separately as part of this case
to enable future work (both in management software and hardware development)
to occur in parallel. These interfaces will initially be delivered as
consolidation private, with the expectation that they will be made public
after we have more experience using them in a wider variety of applications.
SES is an industry standard protocol for describing SCSI device enclosures and
retrieving status information from them. Refer to t10.org for background and
copies of the specifications. SES operates by encoding operations inside of
SCSI packets directed as particular SCSI targets that offer SES services.
All new disk enclosures being produced by Sun, and many by other vendors, use
the SAS protocol as the underlying transport, which in turn escapsulates SCSI,
and therefore encapsulates SES. Thus SES is used to manage a SAS JBOD.
SES was first introduced into Solaris for use with Tabasco back in 1996 as part
of the "envsen" driver, which was renamed to "ses" by a later update.
(Score 1 on the crusty old-timer list if you remember Tabasco. Score 2 if
somewhere in your office right now a Tabasco is collecting dust.)
The ses driver fundamentally provided the ability to statically configure the
driver for a given parent and set of nodes in ses.conf so that it would bind to
these nodes, and then one can use the ioctl API to read and write SES using
some simple C abstractions for the SES data. The binding is described in
ses(7D) and the complete history of this stuff is found in:
PSARC/1996/289 envsen driver
PSARC 1996/421 SES device naming
PSARC/1997/195 Common SCSI Enclosure Services Driver
PSARC 2001/442 Correcting Device Node Type for SES
Later on, Dan "i/o Dan" Price introduced a much more generic mechanism for this
kind of software: the sgen driver. Rather than having to rewrite special kernel
drivers like ses for these operations, sgen is a generic SCSI pass-through
driver that permits userland code to read and write the underlying SCSI
protocol and anything that travels in it, like SES. Notice that if "sgen"
had existed first, no "ses" driver ever would have been needed in Solaris. sgen
exports a generic ioctl API described in uscsi(7I). sgen is described in:
PSARC 1999/525 Generic SCSI Pass Through for Solaris
PSARC 1999/561 Generic SCSI Pass Through -- Corrections and Update
Therefore, the strategy for libses and libscsi, described by this case, is
to provide generic services that can sit on top of uscsi and sgen, entirely
ignoring the legacy "ses" driver. Future work can EOL that driver, but that
is an issue entirely orthogonal to the new software in this case. The project
team requests that new work build on this case, rather than the "ses" driver.
B. LIBSCSI
This library is a thin layer on top of uscsi(7I). It provides an
abstraction for managing actions, consisting of CDB, payload, and sense
data. The library interfaces with devices through a set of engines that
are delivered as separate plugins. While the primary engine for Solaris
is uscsi, the plan is to eventually deliver an emulation engine that
allows for simplified testing or hardware prototyping.
This library also allows cross-platform consumers to be written
generically, by porting the basic libscsi interface but providing a
different engine on different platforms.
The current library header file will be provided in the case materials
as an example of the interfaces.
C. LIBSES
The current ses(7D) driver has several major drawbacks. It is hard to
use, requires specialized kernel support, and is difficult to extend.
Rather than introducing further complexity into the kernel, it is
preferable to construct a library that presents a higher level
interface built on uscsi(7I). This library will allow for future
changes to the spec as well as vendor-specific functionality.
The SES-2 specification presents a straightforward topology of the form:
TARGET -> ENCLOSURE -> AGGREGATE -> ELEMENT
There is one implicit target at the root of the tree, but each
subsequent level can have multiple children. The primary responsibility
of the core libses library is to take snapshots of the device state and
parse the config page into a tree of nodes, where each node has a set of
properties exposed as an nvlist. Control operations are modeled as a
per-node function that takes a string indicating the control operation
and an nvlist of parameters.
Apart from reading the supported pages and parsing the configuration
page, libses itself doesn't know anything about parsing per-node
properties. Instead, these properties are added by plugins, which can
be either framework plugins (SES spec and libses extensions), or
vendor-specific (indexed by vendor/product/revision). Each plugin can
provide the following functionality:
- Descriptions of how to retrieve and parse supported pages.
- Method to fill in properties for a node.
- Method to handle control operations on a node.
This allows plugins to reinterpret existing data (such as the STRING IN
page or vendor-specific enclosure data), parse data from vendor-specific
pages, or define vendor-specific control operations.
The ses2 plugin does the the work of interpreting the SES-2
specification. These properties are prefixed by SES_ (or "ses-" in the
nvlist representation). It provides control operations for setting
properties (a generic control op for a variety of features) and
downloading microcode.
There are also several properties that are not defined by the spec, but
are common across multiple vendors and products. These properties begin
with the prefix LIBSES_ (or "libses-"). The most important one is
LIBSES_EN_PROP_CSN, which is the chassis serial number. The SES spec
doesn't provide a way to correlate multiple SES processes within in a
single logical chassis. In order for SES enumeration of enclosures to
work, we need to know when two different SES targets physically reside
in the same chassis (either accessing the same elements or disparate
elements). Other standard properties include part number and serial
number that can be attached to any element.
The current header files for the library and plugins are provided in
the case materials directory.
D. SESTOPO
While this case does not deliver any public consumers of libscsi or
libses, it does introduce a Private utility, sestopo, which simply
iterates over the SES topology and dumps all properties for all nodes.
This provides a simple test of libses functionality, and a debugging
tool for both enclosure firmware and higher level software. Some
example output:
Node Type: 2
nvlist version: 0
ses-enclosure-id = 0x0
ses-enclosure-service-proc-id = 0x0
ses-enclosure-service-proc-count = 0x0
ses-logical-id = (embedded nvlist)
nvlist version: 0
naa-id-integer = 0x500163600004ddbd
naa-id-type = 0x5
naa-company-id = 0x1636
naa-vendor-specific-a = 0x4ddbd
(end ses-logical-id)
ses-vendor-id = SUN
ses-product-id = StorageTek J4400
ses-product-revision = 1R20
ses-element-index = 0x31
ses-element-type = 0xe
ses-class-description = Enclosure
...
E. DELIVERABLES AND INTERFACES
All the interfaces are Contracted Consolidation Private. While it would be
possible to deliver these objects in a separate package, the core FMA
infrastructure will depend on these libraries, so it seems of little use
to deliver them separately. These devlierables will be part of the core
Solaris packages (SUNWcsl, SUNWhea, SUNWarc). The exception will be
vendor specific plugins, which will be delivered as part of a new
SUNWscsip package in order to facilitate creation of minimized systems
(such as the miniroot) and easier patching.
Summary of Exported Interfaces, all Contracted Consolidation Private:
---------------------------------------------------------------------
/usr/lib/scsi
SCSI libraries and private commands
/usr/lib/scsi/libscsi.so.1
/usr/lib/scsi/{amd64,sparcv9}/libscsi.so.1
SCSI library
/usr/lib/scsi/libses.so.1
/usr/lib/scsi/$MACH64/libses.so.1
SES library
/usr/lib/scsi/plugins/scsi/engines
/usr/lib/scsi/plugins/scsi/engines/uscsi.so
/usr/lib/scsi/plugins/scsi/engines/$MACH64/uscsi.so
SCSI engines and default system engine (uscsi).
/usr/lib/scsi/plugins/ses/framework
/usr/lib/scsi/plugins/ses/framework/libses.so
/usr/lib/scsi/plugins/ses/framework/ses2.so
/usr/lib/scsi/plugins/ses/framework/$MACH64/libses.so
/usr/lib/scsi/plugins/ses/framework/$MACH64/ses2.so
/usr/lib/scsi/plugins/ses/vendor
/usr/lib/scsi/plugins/ses/vendor/Sun-StorageTek-J4400.so
/usr/lib/scsi/plugins/ses/vendor/$MACH64/Sun-StorageTek-J4400.so
SES plugins, including standard framework plugins and a sample
vendor-specific plugin for the J4400.
/usr/lib/scsi/sestopo
Sample SES program for debugging.
/usr/include/scsi
/usr/include/scsi/libscsi.h
/usr/include/scsi/libses.h
SCSI/SES include files
/usr/include/scsi/plugins/ses
/usr/include/scsi/plugins/ses/framework
/usr/include/scsi/plugins/ses/framework/ses2.h
/usr/include/scsi/plugins/ses/framework/libses.h
/usr/include/scsi/plugins/ses/vendor
Plugin-specific headers. The primary purpose is to define
available properties that are exported by a plugin.
--
Mike Shapiro, Sun Microsystems Fishworks. blogs.sun.com/mws/
