On 08/05/14 20:28, Mark Kettenis wrote:
Hi Mark,
Interesting to see sparc64 support in QEMU.
Yeah, it's been a work in progress for quite a while now. There seems to
be two main areas of interest: firstly for people who are now migrating
away from SPARC but need to keep a legacy application(s), and secondly
for open source projects interested in testing across multiple
architectures.
As soon as I step into address 0x1001804 then this is where things start
to go wrong; the TLB (TTE) entry for 0x1800000 which is accessed by %sp
is marked as privileged, but ASI 0x11 is user access only. QEMU's
current behaviour for this is to generate a datafault for the page at
0x1800000 which seems to get all the way through to the retry at the end
of winfixsave, but then hits the breakpoint trap above when executing
the retry.
I've finally located the source of this bug thanks to more testing,
which showed that OpenBSD 4.9 was surprisingly also able to boot
(something I missed this in my original bisection). This allowed me to
track down what was happening fairly easily. The problem is caused by
the fact that 0x1800000 has *two* mappings in the TLB and the way in
which QEMU resolves them.
Compare the state of the TLB when the fill_0_normal trap occurs on
OpenBSD 5.5 (faults, incorrect) and OpenBSD 4.9 (no fault, correct):
OpenBSD 5.5:
(qemu) info tlb
MMU contexts: Primary: 0, Secondary: 0
DMMU dump
...
[14] VA: 1800000, PA: f400000, 4M, priv, RW, locked, ctx 0 local
...
[42] VA: 1800000, PA: f400000, 8k, user, RW, unlocked, ctx 0 local
...
OpenBSD 4.9:
(qemu) info tlb
MMU contexts: Primary: 0, Secondary: 0
DMMU dump
...
[08] VA: 1800000, PA: f400000, 8k, user, RW, unlocked, ctx 0 local
...
[14] VA: 1800000, PA: f400000, 4M, priv, RW, locked, ctx 0 local
...
The bug occurs because the QEMU TLB algorithm currently searches the TLB
*in order* starting from entry 0 until it finds a VA match.
In the OpenBSD 5.5 case, the first mapping it finds is the 4M privileged
mapping, and so the fill_0_normal trap which uses user ASI 0x11 faults
due to not being privileged. This is in contrast to the OpenBSD 4.9 case
where the first mapping it finds is the 8K unprivileged mapping, hence
the fill_0_normal trap succeeds and we proceed to boot.
Does anyone know how real hardware resolves conflicts between multiple
TLB entries with the same VA? My guess would be that the smaller 8K
mapping should take priority, but the documentation in relation to
address aliasing is fairly non-existent so I wondering if there are any
other rules relating to whether privileged mappings should take priority
or not? Once the behaviour is known, it will be fairly easy to fix up
QEMU to match.
It seems that this first hypothesis was incorrect; after some help from
the NetBSD guys we found out that all PROM mappings should default to
privileged. So the issue is no longer to do with the difference between
privileged/unprivileged mappings, but why does the fault occur in the
first place?
I don;t know how the real hardware behaves. But it certainly is the
intention that the 4M "locked" mapping gets used as soon as we've
taken over the trap table. Not sure where the 8K mapping is coming
from.
Finally it does raise an eyebrow that the first window trap taken when
the kernel takes over the trap table is a fill_0_normal *user* trap,
particularly when it's against an *unlocked* TLB entry which could
potentially could have been evicted beforehand. It might be worth
double-checking as to whether this is the intended behaviour or not.
Right. It certainly isn't the intention that we end up a
fill_0_normal at this point. Perhaps %wstate is initialized
differently in QEMU than on real hardware? The OpenBSD bootstrap code
does set %wstate appropriately immediately after taking over the trap
table. We can't really do this earlier since we don't know the
conventions used by the spill and fill handlers provided by the
firmware. But it looks like a Sun Fire T2000 actually initializes
%wstate to 0.
So perhaps we're just getting lucky on real hardware that the prom
code doesn't spill our trap frame and therefore we don't have to fill
it again.
After more work, I believe that your theory here is correct. Take a look
at cpu_initialize() in locore.S:
/*
* Initialize a CPU. This is used both for bootstrapping the first CPU
* and spinning up each subsequent CPU. Basically:
*
* Install trap table.
* Switch to the initial stack.
* Call the routine passed in in cpu_info->ci_spinup.
*/
_C_LABEL(cpu_initialize):
wrpr %g0, 0, %tl ! Make sure we're not in
NUCLEUS mode
flushw
/* Change the trap base register */
set _C_LABEL(trapbase), %l1
#ifdef SUN4V
sethi %hi(_C_LABEL(cputyp)), %l0
ld [%l0 + %lo(_C_LABEL(cputyp))], %l0
cmp %l0, CPU_SUN4V
bne,pt %icc, 1f
nop
set _C_LABEL(trapbase_sun4v), %l1
GET_MMFSA(%o1)
1:
#endif
call _C_LABEL(prom_set_trap_table) ! Now we should be running 100% from
our handlers
mov %l1, %o0
wrpr %l1, 0, %tba ! Make sure the PROM didn't
foul up.
wrpr %g0, WSTATE_KERN, %wstate
My understanding from talking to some of the NetBSD folks is that the
call to prom_set_trap_table() was added later to order to ensure a
"proper" transfer of the trap table in case the PROM was doing some
additional magic. But doing this was what actually introduced the bug in
the first place.
My guess is that originally the code looked like this:
wrpr %l1, 0, %tba
wrpr %g0, WSTATE_KERN, %wstate
This is absolutely fine, because no window fill/spill traps can occur
between these 2 instructions. But introducing the call to
prom_set_trap_table() calls several levels of functions which is what
causes the window fill traps to trigger on return.
The academic part of me would argue that this is a bug, since by calling
"SUNW,set-trap-table" you are declaring that you are taking on
responsibility for handling traps; so the fact that these traps cause a
fatal error means this is not true. Therefore I would suggest that a
more correct version of the code should move the %wstate like this:
wrpr %g0, WSTATE_KERN, %wstate
call _C_LABEL(prom_set_trap_table) ! Now we should be running 100% from
our handlers
mov %l1, %o0
wrpr %l1, 0, %tba ! Make sure the PROM didn't
foul up.
By definition, the PROM cannot determine the value of %wstate when one
of the fill/spill traps occurs since at this point it's already too
late. Hence it has to install its handlers across all window fill/spill
trap levels in order to accommodate this situation (and indeed, both OBP
and OpenBIOS do this).
Based upon this, I believe your statement that you get lucky on real
hardware is most likely to be true. Since OBP enters Forth very early,
it uses possibly just 1 window which doesn't trigger a fill/spill trap
in this critical timeframe. This is in contrast to OpenBIOS which goes
through several levels of C before the Forth engine, and so has much
greater window usage.
The problem at the moment is that even if the above fix were correct and
accepted, it would still make it impossible to run older versions of
OpenBSD which would make the emulation fairly useless. Hence I've
altered the OpenBIOS CIF interface to dump and restore the entire
register window state across CIF calls which allows OpenBSD to start
booting. The mildly annoying part is that this adds quite a bit of
overhead to CIF calls on OpenBIOS, but I can't think of any better
solution for the moment.
Kind regards,
Mark.
