> Date: Thu, 08 May 2014 14:44:30 +0100 > From: Mark Cave-Ayland <mark.cave-ayl...@ilande.co.uk> > > On 06/05/14 19:18, Mark Cave-Ayland wrote:
Hi Mark, Interesting to see sparc64 support in QEMU. > > As soon as I step into address 0x1001804 then this is where things start > > to go wrong; the TLB (TTE) entry for 0x1800000 which is accessed by %sp > > is marked as privileged, but ASI 0x11 is user access only. QEMU's > > current behaviour for this is to generate a datafault for the page at > > 0x1800000 which seems to get all the way through to the retry at the end > > of winfixsave, but then hits the breakpoint trap above when executing > > the retry. > > I've finally located the source of this bug thanks to more testing, > which showed that OpenBSD 4.9 was surprisingly also able to boot > (something I missed this in my original bisection). This allowed me to > track down what was happening fairly easily. The problem is caused by > the fact that 0x1800000 has *two* mappings in the TLB and the way in > which QEMU resolves them. > > Compare the state of the TLB when the fill_0_normal trap occurs on > OpenBSD 5.5 (faults, incorrect) and OpenBSD 4.9 (no fault, correct): > > > OpenBSD 5.5: > > (qemu) info tlb > MMU contexts: Primary: 0, Secondary: 0 > DMMU dump > ... >  VA: 1800000, PA: f400000, 4M, priv, RW, locked, ctx 0 local > ... >  VA: 1800000, PA: f400000, 8k, user, RW, unlocked, ctx 0 local > ... > > OpenBSD 4.9: > > (qemu) info tlb > MMU contexts: Primary: 0, Secondary: 0 > DMMU dump > ... >  VA: 1800000, PA: f400000, 8k, user, RW, unlocked, ctx 0 local > ... >  VA: 1800000, PA: f400000, 4M, priv, RW, locked, ctx 0 local > ... > > > The bug occurs because the QEMU TLB algorithm currently searches the TLB > *in order* starting from entry 0 until it finds a VA match. > > In the OpenBSD 5.5 case, the first mapping it finds is the 4M privileged > mapping, and so the fill_0_normal trap which uses user ASI 0x11 faults > due to not being privileged. This is in contrast to the OpenBSD 4.9 case > where the first mapping it finds is the 8K unprivileged mapping, hence > the fill_0_normal trap succeeds and we proceed to boot. > > Does anyone know how real hardware resolves conflicts between multiple > TLB entries with the same VA? My guess would be that the smaller 8K > mapping should take priority, but the documentation in relation to > address aliasing is fairly non-existent so I wondering if there are any > other rules relating to whether privileged mappings should take priority > or not? Once the behaviour is known, it will be fairly easy to fix up > QEMU to match. I don;t know how the real hardware behaves. But it certainly is the intention that the 4M "locked" mapping gets used as soon as we've taken over the trap table. Not sure where the 8K mapping is coming from. > Finally it does raise an eyebrow that the first window trap taken when > the kernel takes over the trap table is a fill_0_normal *user* trap, > particularly when it's against an *unlocked* TLB entry which could > potentially could have been evicted beforehand. It might be worth > double-checking as to whether this is the intended behaviour or not. Right. It certainly isn't the intention that we end up a fill_0_normal at this point. Perhaps %wstate is initialized differently in QEMU than on real hardware? The OpenBSD bootstrap code does set %wstate appropriately immediately after taking over the trap table. We can't really do this earlier since we don't know the conventions used by the spill and fill handlers provided by the firmware. But it looks like a Sun Fire T2000 actually initializes %wstate to 0. So perhaps we're just getting lucky on real hardware that the prom code doesn't spill our trap frame and therefore we don't have to fill it again.