Re: HYP Kernel boot requirements
On Tue, Nov 26, 2013 at 09:49:10PM +, Santosh Shilimkar wrote: On Tuesday 26 November 2013 12:37 PM, Dave Martin wrote: On Tue, Nov 26, 2013 at 09:47:13AM -0500, Santosh Shilimkar wrote: On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote: On Mon, Nov 25, 2013 at 07:44:08PM +, Santosh Shilimkar wrote: On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: On Mon, Nov 25, 2013 at 04:59:16PM +, Santosh Shilimkar wrote: What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching. Am not challenging the agreed direction but we need to solve the PM problem as well before making all CPU runs boot-loader for HYP kernels as a must have. At least its is a change in boot strategy from existing kernels. Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)? How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode. Is it late to rewrite the OMAP5 firmware? Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used. If the ROM is capable of loading some additional signed Secure World firmware after the ROM itself has booted, PSCI could be implemented in the second, resident firmware payload. Some SoCs ship with boot ROMs that can do that -- is this not the case for OMAP5? On OMAP, the secure devices there is a way to do this but not for general purpose devices. General purpose devices once you exit the ROM code, we exit out of security and only way to re-enter secure word is via ROM implemented monitor/secure APIs. As last resort, you can still write a bunch of instructions acting as warm-bootloader, reserve a page and add them there. Code required is just a shim layer and should not require more than few bytes, after all it has just to check some variables and jump accordingly (+ calling smc if required). Not pretty, agreed, but still better than patching the kernel IMHO (basically you will end up patching the bootloader - and possibly the dtb - instead, to install the shim mentioned above). Lorenzo -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: HYP Kernel boot requirements
On Mon, Nov 25, 2013 at 07:44:08PM +, Santosh Shilimkar wrote: On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: On Mon, Nov 25, 2013 at 04:59:16PM +, Santosh Shilimkar wrote: What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching. Am not challenging the agreed direction but we need to solve the PM problem as well before making all CPU runs boot-loader for HYP kernels as a must have. At least its is a change in boot strategy from existing kernels. Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)? How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode. Is it late to rewrite the OMAP5 firmware? What I meant is that you have the same kernel binary but with two sets of entry points, or maybe a single set of entry points and some global variable that you set during cold boot if the primary CPU is entered in Hyp mode. So you change the boot loader to switch to Hyp before it starts the kernel and leave the additional switching to be done by the secondary entry points (or PM code) based on the variable you set during cold boot. But I would recommend rewriting the firmware. I will look at PSCI more closely and see what can be done here. PSCI has a different secondary start-up and hotplug/suspend protocol and CPUs wake up in Hyp if available. But this requires changing your firmware, PSCI is not a kernel-only thing. Do you need to support both older and newer kernels with the OMAP5 firmware (and possibly same DT)? -- Catalin -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: HYP Kernel boot requirements
On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote: On Mon, Nov 25, 2013 at 07:44:08PM +, Santosh Shilimkar wrote: On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: On Mon, Nov 25, 2013 at 04:59:16PM +, Santosh Shilimkar wrote: What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching. Am not challenging the agreed direction but we need to solve the PM problem as well before making all CPU runs boot-loader for HYP kernels as a must have. At least its is a change in boot strategy from existing kernels. Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)? How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode. Is it late to rewrite the OMAP5 firmware? Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used. What I meant is that you have the same kernel binary but with two sets of entry points, or maybe a single set of entry points and some global variable that you set during cold boot if the primary CPU is entered in Hyp mode. So you change the boot loader to switch to Hyp before it starts the kernel and leave the additional switching to be done by the secondary entry points (or PM code) based on the variable you set during cold boot. But I would recommend rewriting the firmware. I will look at PSCI more closely and see what can be done here. PSCI has a different secondary start-up and hotplug/suspend protocol and CPUs wake up in Hyp if available. But this requires changing your firmware, PSCI is not a kernel-only thing. Do you need to support both older and newer kernels with the OMAP5 firmware (and possibly same DT)? Hmmm.. So thats rules out PSCI for OMAP5 then. Keystone actually has nice way to deal with the problem since the boot-monitor code can be patched up and hence I had no trouble implementing the CPU starts in HYP mode requirement on it. Regards, Santosh -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: HYP Kernel boot requirements
On Tue, Nov 26, 2013 at 09:47:13AM -0500, Santosh Shilimkar wrote: On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote: On Mon, Nov 25, 2013 at 07:44:08PM +, Santosh Shilimkar wrote: On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: On Mon, Nov 25, 2013 at 04:59:16PM +, Santosh Shilimkar wrote: What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching. Am not challenging the agreed direction but we need to solve the PM problem as well before making all CPU runs boot-loader for HYP kernels as a must have. At least its is a change in boot strategy from existing kernels. Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)? How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode. Is it late to rewrite the OMAP5 firmware? Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used. If the ROM is capable of loading some additional signed Secure World firmware after the ROM itself has booted, PSCI could be implemented in the second, resident firmware payload. Some SoCs ship with boot ROMs that can do that -- is this not the case for OMAP5? Cheers ---Dave -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: HYP Kernel boot requirements
On Tuesday 26 November 2013 12:37 PM, Dave Martin wrote: On Tue, Nov 26, 2013 at 09:47:13AM -0500, Santosh Shilimkar wrote: On Tuesday 26 November 2013 09:13 AM, Catalin Marinas wrote: On Mon, Nov 25, 2013 at 07:44:08PM +, Santosh Shilimkar wrote: On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: On Mon, Nov 25, 2013 at 04:59:16PM +, Santosh Shilimkar wrote: What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching. Am not challenging the agreed direction but we need to solve the PM problem as well before making all CPU runs boot-loader for HYP kernels as a must have. At least its is a change in boot strategy from existing kernels. Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)? How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode. Is it late to rewrite the OMAP5 firmware? Well its ROM'ed unfortunately so no choice. OMAP5 ROM did implement a secure API which lets you enter into HYP mode and thats the only thing can be used. If the ROM is capable of loading some additional signed Secure World firmware after the ROM itself has booted, PSCI could be implemented in the second, resident firmware payload. Some SoCs ship with boot ROMs that can do that -- is this not the case for OMAP5? On OMAP, the secure devices there is a way to do this but not for general purpose devices. General purpose devices once you exit the ROM code, we exit out of security and only way to re-enter secure word is via ROM implemented monitor/secure APIs. regards, Santosh -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
HYP Kernel boot requirements [Was ...Re: [PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs]
On Monday 25 November 2013 11:33 AM, Christoffer Dall wrote: On 25 November 2013 08:28, Santosh Shilimkar santosh.shilim...@ti.com wrote: On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote: On 23 November 2013 16:07, Santosh Shilimkar santosh.shilim...@ti.com wrote: Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU. Hence patch the secondary CPU boot path for hyp mode etry. Cc: Marc Zyngier marc.zyng...@arm.com Cc: Christoffer Dall christoffer.d...@linaro.org Cc: Tony Lindgren t...@atomide.com Signed-off-by: Santosh Shilimkar santosh.shilim...@ti.com --- arch/arm/mach-omap2/omap-headsmp.S |7 +++ 1 file changed, 7 insertions(+) diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S index 75e9295..4844dd8 100644 --- a/arch/arm/mach-omap2/omap-headsmp.S +++ b/arch/arm/mach-omap2/omap-headsmp.S @@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM @@ -38,6 +39,12 @@ wait:ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0 +hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /* hmm, this means that currently running this in a guest will fail to bring-up SMP, right? Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ? That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution. Fair enough. Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel? Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode. In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must. I agree, and PSCI is the obvious only correct answer to this. We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas. However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction. What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching. Am not challenging the agreed direction but we need to solve the PM problem as well before making all CPU runs boot-loader for HYP kernels as a must have. At least its is a change in boot strategy from existing kernels. CC'ing few more folks and changing the subject line Regards, Santosh -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: HYP Kernel boot requirements [Was ...Re: [PATCH 2/2] ARM: OMAP5: Add HYP mode entry support for secondary CPUs]
On Mon, Nov 25, 2013 at 04:59:16PM +, Santosh Shilimkar wrote: On Monday 25 November 2013 11:33 AM, Christoffer Dall wrote: On 25 November 2013 08:28, Santosh Shilimkar santosh.shilim...@ti.com wrote: On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote: On 23 November 2013 16:07, Santosh Shilimkar santosh.shilim...@ti.com wrote: Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU. Hence patch the secondary CPU boot path for hyp mode etry. Cc: Marc Zyngier marc.zyng...@arm.com Cc: Christoffer Dall christoffer.d...@linaro.org Cc: Tony Lindgren t...@atomide.com Signed-off-by: Santosh Shilimkar santosh.shilim...@ti.com --- arch/arm/mach-omap2/omap-headsmp.S |7 +++ 1 file changed, 7 insertions(+) diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S index 75e9295..4844dd8 100644 --- a/arch/arm/mach-omap2/omap-headsmp.S +++ b/arch/arm/mach-omap2/omap-headsmp.S @@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM @@ -38,6 +39,12 @@ wait:ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0 +hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /* hmm, this means that currently running this in a guest will fail to bring-up SMP, right? Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ? That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution. Fair enough. Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel? Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode. In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must. I agree, and PSCI is the obvious only correct answer to this. We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas. However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction. What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching. Am not challenging the agreed direction but we need to solve the PM problem as well before making all CPU runs boot-loader for HYP kernels as a must have. At least its is a change in boot strategy from existing kernels. Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)? -- Catalin -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: HYP Kernel boot requirements
On Monday 25 November 2013 12:28 PM, Catalin Marinas wrote: On Mon, Nov 25, 2013 at 04:59:16PM +, Santosh Shilimkar wrote: On Monday 25 November 2013 11:33 AM, Christoffer Dall wrote: On 25 November 2013 08:28, Santosh Shilimkar santosh.shilim...@ti.com wrote: On Monday 25 November 2013 10:09 AM, Christoffer Dall wrote: On 23 November 2013 16:07, Santosh Shilimkar santosh.shilim...@ti.com wrote: Boot-CPU entry into the HYP mode is managed in boot-loader but the secondary CPUs directly jumps to kernel during boot. Same path is also used for CPU hotplug as well during suspend for secondary CPU. Hence patch the secondary CPU boot path for hyp mode etry. Cc: Marc Zyngier marc.zyng...@arm.com Cc: Christoffer Dall christoffer.d...@linaro.org Cc: Tony Lindgren t...@atomide.com Signed-off-by: Santosh Shilimkar santosh.shilim...@ti.com --- arch/arm/mach-omap2/omap-headsmp.S |7 +++ 1 file changed, 7 insertions(+) diff --git a/arch/arm/mach-omap2/omap-headsmp.S b/arch/arm/mach-omap2/omap-headsmp.S index 75e9295..4844dd8 100644 --- a/arch/arm/mach-omap2/omap-headsmp.S +++ b/arch/arm/mach-omap2/omap-headsmp.S @@ -22,6 +22,7 @@ /* Physical address needed since MMU not enabled yet on secondary core */ #define AUX_CORE_BOOT0_PA 0x48281800 +#define API_HYP_ENTRY 0x102 /* * OMAP5 specific entry point for secondary CPU to jump from ROM @@ -38,6 +39,12 @@ wait:ldr r2, =AUX_CORE_BOOT0_PA @ read from AuxCoreBoot0 and r4, r4, #0x0f cmp r0, r4 bne wait +#ifdef CONFIG_KVM_ARM_HOST + ldr r12, =API_HYP_ENTRY + adr r0, hyp_boot + smc #0 +hyp_boot: +#endif b secondary_startup END(omap5_secondary_startup) /* hmm, this means that currently running this in a guest will fail to bring-up SMP, right? Nope. Because the code under 'KVM_ARM_HOST' macro. Guest build will not enable CONFIG_KVM_ARM_HOST and things should be fine then. Right ? That really goes against the whole single binary on all platforms thing. With multi-platform support you really shouldn't have to compile your kernel any differently for running as a guest as when you're running on a host. Someone may even emulate an OMAP5 in QEMU and you'd certainly want your kvm-enabled kernel to run as both guest and host. After all, this is not a paravirtualization solution. Fair enough. Couldn't you create a little wrapper-pen in U-Boot instead, which replicates the omap boot protocol and takes care of the hyp-mode startup there instead, keeping this completely out of the kernel? Its not just booting but CPU hotplug also follows the same path so we need the mechanism in kernel to switch mode. In general, I think its important to consider the aspect with CPU PM. CPUs are not going to go through the boot-loaders in those paths and hence need of HYP entry in the kernel will be must. I agree, and PSCI is the obvious only correct answer to this. We have discussed this a bit earlier (I think Will Deacon brought this up - cc'ed), but I don't think anyone had any bright ideas. However, we broadly agreed on the fact that for KVM/hyp support, you need to boot your kernel in that mode, and this is definitely pulling in the wrong direction. What I am saying is the platforms like OMAP5 already support PM in mainline kernel and we can't break that for KVM. Boot-loaders would be thrashed after boot so you need something which runs in Kernel or along with Kernel to have equivalent of hyp switching. Am not challenging the agreed direction but we need to solve the PM problem as well before making all CPU runs boot-loader for HYP kernels as a must have. At least its is a change in boot strategy from existing kernels. Of course I recommend PSCI which covers both hotplug and suspend ;), but I guess it's not the case for OMAP5. Since OMAP has its own secondary booting protocol and CPU hotplug re-entry, can you not just use different entry point when the primary CPU was initially started in Hyp mode (e.g. omap5_hyp_secondary_startup)? How will that solve the guest secondary boot failure case when using the same kernel binary for guest-boot ? Even for primary CPU which will be suspended it needs to resume already in HYP mode and its not going to go through boot-loader. So the low power code needs to have HYP switch code so that CPU resumes in HYP mode. I will look at PSCI more closely and see what can be done here. Regards, Santosh -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
