On Tue, Sep 27, 2011 at 5:03 PM, Alexander Graf <ag...@suse.de> wrote: > > On 27.09.2011, at 18:53, Blue Swirl wrote: > >> On Tue, Sep 27, 2011 at 3:59 PM, Alexander Graf <ag...@suse.de> wrote: >>> >>> On 27.09.2011, at 17:50, Blue Swirl wrote: >>> >>>> On Mon, Sep 26, 2011 at 11:19 PM, Scott Wood <scottw...@freescale.com> >>>> wrote: >>>>> On 09/24/2011 05:00 AM, Alexander Graf wrote: >>>>>> On 24.09.2011, at 10:44, Blue Swirl wrote: >>>>>>> On Sat, Sep 24, 2011 at 8:03 AM, Alexander Graf <ag...@suse.de> wrote: >>>>>>>> On 24.09.2011, at 09:41, Blue Swirl wrote: >>>>>>>>> On Mon, Sep 19, 2011 at 4:12 PM, Scott Wood <scottw...@freescale.com> >>>>>>>>> wrote: >>>>>>>>>> The goal with the spin table stuff, suboptimal as it is, was >>>>>>>>>> something >>>>>>>>>> that would work on any powerpc implementation. Other >>>>>>>>>> implementation-specific release mechanisms are allowed, and are >>>>>>>>>> indicated by a property in the cpu node, but only if the loader knows >>>>>>>>>> that the OS supports it. >>>>>>>>>> >>>>>>>>>>> IIUC the spec that includes these bits is not finalized yet. It is >>>>>>>>>>> however in use on all u-boot versions for e500 that I'm aware of >>>>>>>>>>> and the method Linux uses to bring up secondary CPUs. >>>>>>>>>> >>>>>>>>>> It's in ePAPR 1.0, which has been out for a while now. ePAPR 1.1 was >>>>>>>>>> just released which clarifies some things such as WIMG. >>>>>>>>>> >>>>>>>>>>> Stuart / Scott, do you have any pointers to documentation where the >>>>>>>>>>> spinning is explained? >>>>>>>>>> >>>>>>>>>> https://www.power.org/resources/downloads/Power_ePAPR_APPROVED_v1.1.pdf >>>>>>>>> >>>>>>>>> Chapter 5.5.2 describes the table. This is actually an interface >>>>>>>>> between OS and Open Firmware, obviously there can't be a real hardware >>>>>>>>> device that magically loads r3 etc. >>>>> >>>>> Not Open Firmware, but rather an ePAPR-compliant loader. >>>> >>>> 'boot program to client program interface definition'. >>>> >>>>>>>>> The device method would break abstraction layers, >>>>> >>>>> Which abstraction layers? >>>> >>>> QEMU system emulation emulates hardware, not software. Hardware >>>> devices don't touch CPU registers. >>> >>> The great part about this emulated device is that it's basically guest >>> software running in host context. To the guest, it's not a device in the >>> ordinary sense, such as vmport, but rather the same as software running on >>> another core, just that the other core isn't running any software. >>> >>> Sure, if you consider this a device, it does break abstraction layers. Just >>> consider it as host running guest code, then it makes sense :). >>> >>>> >>>>>>>>> it's much like >>>>>>>>> vmport stuff in x86. Using a hypercall would be a small improvement. >>>>>>>>> Instead it should be possible to implement a small boot ROM which puts >>>>>>>>> the secondary CPUs into managed halt state without spinning, then the >>>>>>>>> boot CPU could send an IPI to a halted CPU to wake them up based on >>>>>>>>> the spin table, just like real HW would do. >>>>> >>>>> The spin table, with no IPI or halt state, is what real HW does (or >>>>> rather, what software does on real HW) today. It's ugly and inefficient >>>>> but it should work everywhere. Anything else would be dependent on a >>>>> specific HW implementation. >>>> >>>> Yes. Hardware doesn't ever implement the spin table. >>>> >>>>>>>>> On Sparc32 OpenBIOS this >>>>>>>>> is something like a few lines of ASM on both sides. >>>>>>>> >>>>>>>> That sounds pretty close to what I had implemented in v1. Back then >>>>>>>> the only comment was to do it using this method from Scott. >>>>> >>>>> I had some comments on the actual v1 implementation as well. :-) >>>>> >>>>>>>> So we have the choice between having code inside the guest that >>>>>>>> spins, maybe even only checks every x ms, by programming a timer, >>>>>>>> or we can try to make an event out of the memory write. V1 was >>>>>>>> the former, v2 (this one) is the latter. This version performs a >>>>>>>> lot better and is easier to understand. >>>>>>> >>>>>>> The abstraction layers should not be broken lightly, I suppose some >>>>>>> performance or laziness^Wlocal optimization reasons were behind vmport >>>>>>> design too. The ideal way to solve this could be to detect a spinning >>>>>>> CPU and optimize that for all architectures, that could be tricky >>>>>>> though (if a CPU remains in the same TB for extended periods, inspect >>>>>>> the TB: if it performs a loop with a single load instruction, replace >>>>>>> the load by a special wait operation for any memory stores to that >>>>>>> page). >>>>> >>>>> How's that going to work with KVM? >>>>> >>>>>> In fact, the whole kernel loading way we go today is pretty much >>>>>> wrong. We should rather do it similar to OpenBIOS where firmware >>>>>> always loads and then pulls the kernel from QEMU using a PV >>>>>> interface. At that point, we would have to implement such an >>>>>> optimization as you suggest. Or implement a hypercall :). >>>>> >>>>> I think the current approach is more usable for most purposes. If you >>>>> start U-Boot instead of a kernel, how do pass information on from the >>>>> user (kernel, rfs, etc)? Require the user to create flash images[1]? >>>> >>>> No, for example OpenBIOS gets the kernel command line from fw_cfg device. >>>> >>>>> Maybe that's a useful mode of operation in some cases, but I don't think >>>>> we should be slavishly bound to it. Think of the current approach as >>>>> something between whole-system and userspace emulation. >>>> >>>> This is similar to ARM, M68k and Xtensa semi-hosting mode, but not at >>>> kernel level but lower. Perhaps this mode should be enabled with >>>> -semihosting flag or a new flag. Then the bare metal version could be >>>> run without the flag. >>> >>> and then we'd have 2 implementations for running in system emulation mode >>> and need to maintain both. I don't think that scales very well. >> >> No, but such hacks are not common. >> >>>> >>>>> Where does the device tree come from? How do you tell the guest about >>>>> what devices it has, especially in virtualization scenarios with non-PCI >>>>> passthrough devices, or custom qdev instantiations? >>>>> >>>>>> But at least we'd always be running the same guest software stack. >>>>> >>>>> No we wouldn't. Any U-Boot that runs under QEMU would have to be >>>>> heavily modified, unless we want to implement a ton of random device >>>>> emulation, at least one extra memory translation layer (LAWs, localbus >>>>> windows, CCSRBAR, and such), hacks to allow locked cache lines to >>>>> operate despite a lack of backing store, etc. >>>> >>>> I'd say HW emulation business as usual. Now with the new memory API, >>>> it should be possible to emulate the caches with line locking and TLBs >>>> etc., this was not previously possible. IIRC implementing locked cache >>>> lines would allow x86 to boot unmodified coreboot. >>> >>> So how would you emulate cache lines with line locking on KVM? >> >> The cache would be a MMIO device which registers to handle all memory >> space. Configuring the cache controller changes how the device >> operates. Put this device between CPU and memory and other devices. >> Performance would probably be horrible, so CPU should disable the >> device automatically after some time. > > So how would you execute code on this region then? :)
Easy, fix QEMU to allow executing from MMIO. (Yeah, I forgot about that). >> >>> However, we already have a number of hacks in SeaBIOS to run in QEMU, so I >>> don't see an issue in adding a few here and there in u-boot. The memory >>> pressure is a real issue though. I'm not sure how we'd manage that one. >>> Maybe we could try and reuse the host u-boot binary? heh >> >> I don't think SeaBIOS breaks layering except for fw_cfg. > > I'm not saying we're breaking layering there. I'm saying that changing u-boot > is not so bad, since it's the same as we do with SeaBIOS. It was an argument > in favor of your position. Never mind then ;-) >> For extremely >> memory limited situation, perhaps QEMU (or Native KVM Tool for lean >> and mean version) could be run without glibc, inside kernel or even >> interfacing directly with the hypervisor. I'd also continue making it >> possible to disable building unused devices and features. > > I'm pretty sure you're not the only one with that goal ;). Great, let's do it.
