The fast and compact way to document the Solaris/PPC 2.11
HAT layer is to start with the existing Solaris/PPC 2.6
code and highlight the differences. After all, this is
not a "virgin" porting job; we are re-porting Solaris,
and the 2.6 code was used as the starting point for our
modifications. Describing those modifications is all that
I would do, if I were doing an internal hand-off to some
other Sun programmers, and nothing more.
For the sake of a wider audience, I think it appropriate to
make a few comments about VM and HAT layer implementations,
in general, and how sausage is made, and to provide a bit
of an overview of Solaris/PPC 2.6 HAT code in particular.
Then, we can move on to a description of the changes I
have made.
Minimal HAT changes
-------------------
Very little of the time spent on Solaris/PPC 2.11 was
devoted to working on HAT code. This may come as a bit
of a surprise to some. It certainly was not the plan of
record, when I started.
That is good news or bad news, depending on your point
of view.
The good news is two-fold:
1. The existing code was in good enough shape
to get us up and running.
2. It means there are not really many differences
to describe.
The bad news is there is still a big list of missing
features:
o No multi-processor (MP)
o No 64 bit
o No intimate shared memory (ISM)
For some, it could mean that you have not missed out,
because there is still a great deal of freedom of choice
for anyone wanting to take over and work on extensions or
improved replacement code.
So little work was done on the Solaris/PPC HAT layer
for two reasons:
1. The existing code was "good enough";
2. The project was budgeted and scheduled
in small increments. And HAT layer
development is path-dependent.
Reason #1
---------
Because the HAT code was workable and so many other
things were simply not being done, work on the HAT code
took a back seat to things like locore, mlsetup, startup
(physical memory management, kernel address space layout,
etc.), traps, Poor Man's DeBugger (PMDB), VOF, inetboot,
locking primitives, even low-level functions like bzero,
bcopy, etc.; upgrading from 2.6 to 2.11; libraries,
particularly libc, build tools, ODW board issues, and so on.
Reason #2
---------
It is an imperfect world. The approach taken to work on
a task like a Solaris port really is influenced heavily,
if not completely determined, by the amount of time
and money available. And, not just the total amount
of time and money -- even the way the same total amount
of resources are distributed over time, the distribution
of deadlines, and the certainty of budget and schedule,
and the necessity of managing expectations along the way.
I know I would take a different approach entirely, not
only to the HAT layer, but to the entire task of porting
Solaris -- to PowerPC or to any other processor -- if I did
not have to take all these other things into consideration.
I suppose that sounds like a pretty silly thing to say.
Anyone would like infinite time and money. That's not
saying anything we don't all know, already. What I am
saying is that, even if you take a finite amount of money
and see what kind of results you get in the end, after
going through the incremental and uncertain process of
development, then take the same amount of money and ask
me to just go away and write a HAT layer on a fixed cost
basis, the approach I would take would be quite different,
given the certainty of commitment and budget and the lack
of need to manage expectations during the whole process.
My involvement started with a very short-term contract
to write up a feasibility study for porting the HAT
layer to new hardware and to a new release of Solaris.
Nothing more. As things progressed, my time horizon was
never more than two months.
The schedule and the nature of the work done on the
Solaris/PPC HAT layer was heavily influenced by the
need to always be ready to walk away, on short notice.
It meant that no really ambitious changes were made.
Only a series of small, safe transformation were applied.
You may think I am complaining. Well, yes and no.
This project is not unique in that respect. It seems to
be a fact of corporate life, and of software development,
in general. Perhaps, as an Open Source project, the
Polaris HAT layer, and for that matter, just about all
other aspects of porting, could be managed with a Patrician
disregard for time and money, and so the community could
do the port the right way.
A lot of software development is path-dependent. I think
HAT layer development is especially so, partly for reasons
that are purely technical, and partly for "contextual" reasons;
that is, other real-world commercial factors, perceptions,
historical reasons.
Where is "HAT for Dummies" ?
----------------------------
Historically, there has not been that much information
published about HAT code -- not compared to just about
any other kind of software, including operating system
internals. One reason is that it is processor-specific,
even platform-specific, in some cases. If you survey the
textbooks available in university courses over the last
decade or so, you will see books that cover all sorts of
things: filesystems, networking, processes, threads, even
virtual memory, but mostly generic (processor-independent).
Sun Education offers a course on Solaris internals.
You will find there coverage of virtual memory: processes,
address spaces, segments, the kernel memory allocator, etc.
And, there are descriptions of physical memory management:
page_t's, page coloring, and so on. And then, you can
get descriptions of the MMU hardware for various machines.
But, in-between ... uhhm, well ... a miracle occurs.
The main reason for this lack of coverage is simply that
the HAT layer is specific to each processor/platform.
A lot of other processor-specific gets short shrift in
the text books and courses.
For HAT code, things have been getting better, lately.
For instance, the second edition of Solaris Internals has
much more coverage of the HAT layer than the first edition.
The obscurity spiral
--------------------
Because HAT layer is inherently processor-specific,
the market for it is thin. There are relatively
few people who actually have experience designing
and implementing operating system internals. It is
more so for all platform-specific code, even more so
for platform-specific VM, and _especially_ so for
the HAT layer. All of this combines to make HAT code,
historically, somewhat obscure. And so, it gets little
coverage in textbooks and courses, and that makes it
more obscure.
Writing a HAT layer is not an easy job. And, the HAT
layer is critical. As if that were not enough, these
other factors compound the problems for any company that
wants to manage a port of an operating system.
The long pole
-------------
Why is there such a big deal made of the HAT layer? I said
it is critical. What exactly does that mean? Well, if
it does not work correctly, the entire project is lost.
But other things are critical. Filesystem code has to
work correctly.
OK, so it is not just that it is critical. It is critical,
AND it is processor-specific, so it cannot possibly be
just a matter of recompiling common code, whereas, if it
is done with some precautions in mind in the first place,
filesystem code can and ought to be just that simple to port.
Another thing about the HAT layer is that it comes into
play early on. Some portions having to do with user-land
may not have to be working right away, but a quite a bit
of functionality has to be ready, very early in the life
of a porting project.
So, 1) it is critical, 2) much code has to be written for
any port, and 3) kernel HAT has to be brought up quickly.
For these reasons, the HAT layer is generally thought
of as "the long pole in the tent". That term has seen
some abuse, so I will use this traditional engineering
definition:
"the thing that must be dealt with first or most
crucially for a larger project to be successful."
In the commercial OS world, at least, a port of
an operating system requires many programmers.
Sun has been known to do it with about a dozen, or so,
for "new" processors. I don't have the figures for the
port to AMD64. The port to IA64 was 12 people. And that
is lean and mean by corporate standards. Intel and IBM
used to complain that Sun is not committed to the IA64,
because a mere 12 people worked on it, whereas IBM
had over 100, and there were similar commitments from
Hewlett-Packard, and of course Intel, not even counting
the hardware side of things.
What this means is that, if the HAT layer does not come up
soon, and reasonably bug-free, 11 (or more) other people
are held up, waiting for you. They don't call it "the
hot seat" for nothing.
So, it is not just inadequate funding, or staff,
or lack of certainty or lack of commitment that can
profoundly influence the development path of HAT code.
In a well-funded project for a "real product", where there
is no question about commitment or project duration or
other resources, the order in which things are implemented,
and the methods and tools used, are pretty much dictated
by the requirements of others, and are path-dependent.
Because HAT functionality is needed in stages, and because
you really cannot afford to hold up all those other people,
the HAT layer really does (almost always) get developed
in stages, and successive refinements. It is not one of
those cases where it just gets presented that way for
teaching purposes. It really is done that way. And,
it is not because the person doing the job didn't know
enough to do it right in the first place. A developer
with previous experience porting an OS and writing HAT
code would still do things in stages, keeping one step
ahead of the other members of the porting team.
So, you have your choice: either the project is
under-funded, so you warp the development schedule from the
"natural" path of development due to lack of resources;
or the project is very well funded, well staffed, and
otherwise well provisioned and well run, in which case,
there are all those other people depending on you, and
you warp the development schedule.
Note: I think with the advent of things like mdb/kmdb
and Dtrace, the trend is clearly toward higher porting
costs, due to their inherently processor-specific nature.
However, we were able to get things up and running without
them, but that cannot be said for the HAT.
Conceptually simple
-------------------
"All programming is an exercise in caching."
-- Terje Marthisen
After all that talk about how critical the HAT layer
is, it helps to keep in mind that the job of the HAT
layer is conceptually very simple. It is just a cache.
It is just that it is a cache of address translations,
not of memory contents. Like other forms of caching,
it is conceptually simple, but difficult to get right.
One possible way of studying HAT design and implementation
would be to write a toy HAT that is not required to
do more than the minimum, but to function correctly.
This is easier on some hardware than on others. Then,
more functionality can be added. Since this is pretty
much what I described as the process for brand new ports,
it turns out that there is no _fundamental_ conflict
between what is good for pedagogical purposes and the
way things are developed under time constraints. So,
I would recommend that approach for getting started.
Just do it on your own the first time, not in the hot seat.
The difference between an experienced HAT developer and
a dilettante is NOT than one writes a complete, fully
formed HAT in one pass, while the other has to write
several iterations. No, they both do things in stages.
The professional knows ahead of time that things will
be done in stages, and knows how to plan ahead for that,
and not paint himself into a corner.
-- Guy Shaw
