Re: [Xen-devel] [PATCH 6/7] xen/arm: Setup virtual paging for secondary CPUs in non-boot scenario
On 18/04/18 11:45, Mirela Simonovic wrote: On Tue, Apr 17, 2018 at 4:11 PM, Julien Grallwrote: On 17/04/18 13:54, Mirela Simonovic wrote: Hi Julien, Hi, On Wed, Apr 11, 2018 at 5:11 PM, Julien Grall wrote: Hi, On 11/04/18 14:19, Mirela Simonovic wrote: In existing code the paging for secondary CPUs is setup only in boot flow. The setup is triggered from start_xen function after all CPUs are brought online. In other words, the initialization of VTCR_EL2 register is done out of the cpu_up/start_secondary control flow. However, the cpu_up flow should be self-contained - it should fully initialize a secondary CPU, because the cpu_up is used not only to bring a secondary CPU online on boot, but also to hotplug a CPU during the system resume. With this patch the setting of paging is triggered from start_secondary function if the current system state is not boot. This way, the paging will be setup in non-boot scenarios, while the setup in boot scenario remains unchanged. I am afraid that this is not correct. You can't assume that value chosen for VTCR by Xen at boot will fit this new CPU. So you have to check it is fine or park the CPU if there are any issue. This is not a new CPU. This CPU already went through its boot sequence and it reached the resume point because it does fit the value chosen for VTCR by Xen. If it wouldn't fit the chosen value for VTCR it would be parked so it wouldn't participate in suspend/resume. Please let me know if I misunderstood your comment. This is not a new CPU for your use case. However your commit message speak about "non-boot" CPU bring-up. So for me this is more than suspend/resume, it is about bringing-up CPU at any time. As those CPUs can't participate to the decision (it is too late), you need to make sure the VTCR will fit on that CPU. AFAIU the value chosen by Xen for VTCR config has to be common for all online CPUs. Since this value is also used in the resume path I suggest to make global (static in the p2m.c) the 'val' variable which is currently local in setup_virt_paging() and passed as argument to setup_virt_paging_one(). Then setup_virt_paging_one() would not receive an argument. I need to access this value on resume, so I would call setup_virt_paging_one() without argument from start_secondary() if the system state is not boot. This seems to me a bit cleaner compared to what I submitted in this patch, but fundamentally the functionality is the same. You don't need to introduce a static variable it. I believe you can re-create it based on the information we already have in global variables. So what I would do is moving the creation of vtcr value in that function. Using this approach each CPU will need to recalculate the value which is already known prior to executing the function. I believe this is sub-optimal and contrary to existing implementation where only one CPU performs the calculation. The implementation usually evolves with the requirements. In the existing implementation, the value could be calculated on a single pCPU and then scratched afterwards. Is there any benefit of recalculating the value? > Is there any disadvantage of remembering the value into a static variable? I know that it is only a 32-bit value, but I would rather avoid spreading static variable when a value can be recompute in a few steps with what we have. Stefano do you have any opinions? Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 6/7] xen/arm: Setup virtual paging for secondary CPUs in non-boot scenario
On Tue, Apr 17, 2018 at 4:11 PM, Julien Grallwrote: > > > On 17/04/18 13:54, Mirela Simonovic wrote: >> >> Hi Julien, > > > Hi, > >> >> On Wed, Apr 11, 2018 at 5:11 PM, Julien Grall >> wrote: >>> >>> Hi, >>> >>> On 11/04/18 14:19, Mirela Simonovic wrote: In existing code the paging for secondary CPUs is setup only in boot flow. The setup is triggered from start_xen function after all CPUs are brought online. In other words, the initialization of VTCR_EL2 register is done out of the cpu_up/start_secondary control flow. However, the cpu_up flow should be self-contained - it should fully initialize a secondary CPU, because the cpu_up is used not only to bring a secondary CPU online on boot, but also to hotplug a CPU during the system resume. With this patch the setting of paging is triggered from start_secondary function if the current system state is not boot. This way, the paging will be setup in non-boot scenarios, while the setup in boot scenario remains unchanged. >>> >>> >>> >>> I am afraid that this is not correct. You can't assume that value chosen >>> for >>> VTCR by Xen at boot will fit this new CPU. So you have to check it is >>> fine >>> or park the CPU if there are any issue. >>> >> >> This is not a new CPU. This CPU already went through its boot sequence >> and it reached the resume point because it does fit the value chosen >> for VTCR by Xen. >> If it wouldn't fit the chosen value for VTCR it would be parked so it >> wouldn't participate in suspend/resume. Please let me know if I >> misunderstood your comment. > > > This is not a new CPU for your use case. However your commit message > speak about "non-boot" CPU bring-up. So for me this is more than > suspend/resume, it is about bringing-up CPU at any time. > > As those CPUs can't participate to the decision (it is too late), you > need to make sure the VTCR will fit on that CPU. > >> >> AFAIU the value chosen by Xen for VTCR config has to be common for all >> online CPUs. Since this value is also used in the resume path I >> suggest to make global (static in the p2m.c) the 'val' variable which >> is currently local in setup_virt_paging() and passed as argument to >> setup_virt_paging_one(). Then setup_virt_paging_one() would not >> receive an argument. >> I need to access this value on resume, so I would call >> setup_virt_paging_one() without argument from start_secondary() if the >> system state is not boot. >> This seems to me a bit cleaner compared to what I submitted in this >> patch, but fundamentally the functionality is the same. > > > You don't need to introduce a static variable it. I believe you can > re-create it based on the information we already have in global > variables. So what I would do is moving the creation of vtcr value in > that function. Using this approach each CPU will need to recalculate the value which is already known prior to executing the function. I believe this is sub-optimal and contrary to existing implementation where only one CPU performs the calculation. Is there any benefit of recalculating the value? Is there any disadvantage of remembering the value into a static variable? Please let me know, I just need to understand whether there is any particular reason why you want to repeat the calculation. If no particular reasons, saving the calculated value is definitely a better approach. Thanks, Mirela > > Cheers, > > -- > Julien Grall > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 6/7] xen/arm: Setup virtual paging for secondary CPUs in non-boot scenario
Hi Julien, On Wed, Apr 18, 2018 at 11:48 AM, Julien Grallwrote: > > > On 17/04/18 16:22, Mirela Simonovic wrote: >> >> Hi Julien, >> >> On Tue, Apr 17, 2018 at 4:11 PM, Julien Grall >> wrote: >>> >>> >>> >>> On 17/04/18 13:54, Mirela Simonovic wrote: Hi Julien, >>> >>> >>> >>> Hi, >>> On Wed, Apr 11, 2018 at 5:11 PM, Julien Grall wrote: > > > Hi, > > On 11/04/18 14:19, Mirela Simonovic wrote: >> >> >> >> In existing code the paging for secondary CPUs is setup only in boot >> flow. >> The setup is triggered from start_xen function after all CPUs are >> brought >> online. In other words, the initialization of VTCR_EL2 register is >> done >> out of the cpu_up/start_secondary control flow. However, the cpu_up >> flow >> should be self-contained - it should fully initialize a secondary CPU, >> because the cpu_up is used not only to bring a secondary CPU online on >> boot, but also to hotplug a CPU during the system resume. >> With this patch the setting of paging is triggered from >> start_secondary >> function if the current system state is not boot. This way, the paging >> will be setup in non-boot scenarios, while the setup in boot scenario >> remains unchanged. > > > > > I am afraid that this is not correct. You can't assume that value > chosen > for > VTCR by Xen at boot will fit this new CPU. So you have to check it is > fine > or park the CPU if there are any issue. > This is not a new CPU. This CPU already went through its boot sequence and it reached the resume point because it does fit the value chosen for VTCR by Xen. If it wouldn't fit the chosen value for VTCR it would be parked so it wouldn't participate in suspend/resume. Please let me know if I misunderstood your comment. >>> >>> >>> >>> This is not a new CPU for your use case. However your commit message >>> speak about "non-boot" CPU bring-up. So for me this is more than >>> suspend/resume, it is about bringing-up CPU at any time. >>> >> >> Use case you're trying to cover is hotplugging a CPU after the boot is >> done in bit.LITTLE system, and that CPU wasn't initially brought >> online (on boot). Right? > > > That's right. It is how I understood your commit title. > >> >>> As those CPUs can't participate to the decision (it is too late), you >>> need to make sure the VTCR will fit on that CPU. >>> >> >> Could you please point me to the location in sources where this is >> done on boot? I mean checking compliance with chosen VTCR value and >> parking CPU if it doesn't fit. > > > At the moment all CPUs are brought up during Xen boot. So what we do is find > the small PA range that will accommodate all the onlined CPUs (see > setup_virt_paging). > > So there are no parking required at the moment. However, if you start > bringing-up new CPUs after boot. Then you have to check they will be able to > handle the values chosen at boot. > > As you pointed out, this might not be necessary for the suspend/resume case. > So I am not asking you to implement that. However, please make sure the > commit message clearly spell out that you bringing up secondary CPU after > Xen has boot only covers the suspend/resume case. > I'll fix the commit title, thanks. > Cheers, > > -- > Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 6/7] xen/arm: Setup virtual paging for secondary CPUs in non-boot scenario
On 17/04/18 16:22, Mirela Simonovic wrote: Hi Julien, On Tue, Apr 17, 2018 at 4:11 PM, Julien Grallwrote: On 17/04/18 13:54, Mirela Simonovic wrote: Hi Julien, Hi, On Wed, Apr 11, 2018 at 5:11 PM, Julien Grall wrote: Hi, On 11/04/18 14:19, Mirela Simonovic wrote: In existing code the paging for secondary CPUs is setup only in boot flow. The setup is triggered from start_xen function after all CPUs are brought online. In other words, the initialization of VTCR_EL2 register is done out of the cpu_up/start_secondary control flow. However, the cpu_up flow should be self-contained - it should fully initialize a secondary CPU, because the cpu_up is used not only to bring a secondary CPU online on boot, but also to hotplug a CPU during the system resume. With this patch the setting of paging is triggered from start_secondary function if the current system state is not boot. This way, the paging will be setup in non-boot scenarios, while the setup in boot scenario remains unchanged. I am afraid that this is not correct. You can't assume that value chosen for VTCR by Xen at boot will fit this new CPU. So you have to check it is fine or park the CPU if there are any issue. This is not a new CPU. This CPU already went through its boot sequence and it reached the resume point because it does fit the value chosen for VTCR by Xen. If it wouldn't fit the chosen value for VTCR it would be parked so it wouldn't participate in suspend/resume. Please let me know if I misunderstood your comment. This is not a new CPU for your use case. However your commit message speak about "non-boot" CPU bring-up. So for me this is more than suspend/resume, it is about bringing-up CPU at any time. Use case you're trying to cover is hotplugging a CPU after the boot is done in bit.LITTLE system, and that CPU wasn't initially brought online (on boot). Right? That's right. It is how I understood your commit title. As those CPUs can't participate to the decision (it is too late), you need to make sure the VTCR will fit on that CPU. Could you please point me to the location in sources where this is done on boot? I mean checking compliance with chosen VTCR value and parking CPU if it doesn't fit. At the moment all CPUs are brought up during Xen boot. So what we do is find the small PA range that will accommodate all the onlined CPUs (see setup_virt_paging). So there are no parking required at the moment. However, if you start bringing-up new CPUs after boot. Then you have to check they will be able to handle the values chosen at boot. As you pointed out, this might not be necessary for the suspend/resume case. So I am not asking you to implement that. However, please make sure the commit message clearly spell out that you bringing up secondary CPU after Xen has boot only covers the suspend/resume case. Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 6/7] xen/arm: Setup virtual paging for secondary CPUs in non-boot scenario
Hi Julien, On Tue, Apr 17, 2018 at 4:11 PM, Julien Grallwrote: > > > On 17/04/18 13:54, Mirela Simonovic wrote: >> >> Hi Julien, > > > Hi, > >> >> On Wed, Apr 11, 2018 at 5:11 PM, Julien Grall >> wrote: >>> >>> Hi, >>> >>> On 11/04/18 14:19, Mirela Simonovic wrote: In existing code the paging for secondary CPUs is setup only in boot flow. The setup is triggered from start_xen function after all CPUs are brought online. In other words, the initialization of VTCR_EL2 register is done out of the cpu_up/start_secondary control flow. However, the cpu_up flow should be self-contained - it should fully initialize a secondary CPU, because the cpu_up is used not only to bring a secondary CPU online on boot, but also to hotplug a CPU during the system resume. With this patch the setting of paging is triggered from start_secondary function if the current system state is not boot. This way, the paging will be setup in non-boot scenarios, while the setup in boot scenario remains unchanged. >>> >>> >>> >>> I am afraid that this is not correct. You can't assume that value chosen >>> for >>> VTCR by Xen at boot will fit this new CPU. So you have to check it is >>> fine >>> or park the CPU if there are any issue. >>> >> >> This is not a new CPU. This CPU already went through its boot sequence >> and it reached the resume point because it does fit the value chosen >> for VTCR by Xen. >> If it wouldn't fit the chosen value for VTCR it would be parked so it >> wouldn't participate in suspend/resume. Please let me know if I >> misunderstood your comment. > > > This is not a new CPU for your use case. However your commit message > speak about "non-boot" CPU bring-up. So for me this is more than > suspend/resume, it is about bringing-up CPU at any time. > Use case you're trying to cover is hotplugging a CPU after the boot is done in bit.LITTLE system, and that CPU wasn't initially brought online (on boot). Right? > As those CPUs can't participate to the decision (it is too late), you > need to make sure the VTCR will fit on that CPU. > Could you please point me to the location in sources where this is done on boot? I mean checking compliance with chosen VTCR value and parking CPU if it doesn't fit. Thanks, Mirela >> >> AFAIU the value chosen by Xen for VTCR config has to be common for all >> online CPUs. Since this value is also used in the resume path I >> suggest to make global (static in the p2m.c) the 'val' variable which >> is currently local in setup_virt_paging() and passed as argument to >> setup_virt_paging_one(). Then setup_virt_paging_one() would not >> receive an argument. >> I need to access this value on resume, so I would call >> setup_virt_paging_one() without argument from start_secondary() if the >> system state is not boot. >> This seems to me a bit cleaner compared to what I submitted in this >> patch, but fundamentally the functionality is the same. > > > You don't need to introduce a static variable it. I believe you can > re-create it based on the information we already have in global > variables. So what I would do is moving the creation of vtcr value in > that function. > > Cheers, > > -- > Julien Grall > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 6/7] xen/arm: Setup virtual paging for secondary CPUs in non-boot scenario
On 17/04/18 13:54, Mirela Simonovic wrote: Hi Julien, Hi, On Wed, Apr 11, 2018 at 5:11 PM, Julien Grallwrote: Hi, On 11/04/18 14:19, Mirela Simonovic wrote: In existing code the paging for secondary CPUs is setup only in boot flow. The setup is triggered from start_xen function after all CPUs are brought online. In other words, the initialization of VTCR_EL2 register is done out of the cpu_up/start_secondary control flow. However, the cpu_up flow should be self-contained - it should fully initialize a secondary CPU, because the cpu_up is used not only to bring a secondary CPU online on boot, but also to hotplug a CPU during the system resume. With this patch the setting of paging is triggered from start_secondary function if the current system state is not boot. This way, the paging will be setup in non-boot scenarios, while the setup in boot scenario remains unchanged. I am afraid that this is not correct. You can't assume that value chosen for VTCR by Xen at boot will fit this new CPU. So you have to check it is fine or park the CPU if there are any issue. This is not a new CPU. This CPU already went through its boot sequence and it reached the resume point because it does fit the value chosen for VTCR by Xen. If it wouldn't fit the chosen value for VTCR it would be parked so it wouldn't participate in suspend/resume. Please let me know if I misunderstood your comment. This is not a new CPU for your use case. However your commit message speak about "non-boot" CPU bring-up. So for me this is more than suspend/resume, it is about bringing-up CPU at any time. As those CPUs can't participate to the decision (it is too late), you need to make sure the VTCR will fit on that CPU. AFAIU the value chosen by Xen for VTCR config has to be common for all online CPUs. Since this value is also used in the resume path I suggest to make global (static in the p2m.c) the 'val' variable which is currently local in setup_virt_paging() and passed as argument to setup_virt_paging_one(). Then setup_virt_paging_one() would not receive an argument. I need to access this value on resume, so I would call setup_virt_paging_one() without argument from start_secondary() if the system state is not boot. This seems to me a bit cleaner compared to what I submitted in this patch, but fundamentally the functionality is the same. You don't need to introduce a static variable it. I believe you can re-create it based on the information we already have in global variables. So what I would do is moving the creation of vtcr value in that function. Cheers, -- Julien Grall IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 6/7] xen/arm: Setup virtual paging for secondary CPUs in non-boot scenario
Hi Julien, On Wed, Apr 11, 2018 at 5:11 PM, Julien Grallwrote: > Hi, > > On 11/04/18 14:19, Mirela Simonovic wrote: >> >> In existing code the paging for secondary CPUs is setup only in boot flow. >> The setup is triggered from start_xen function after all CPUs are brought >> online. In other words, the initialization of VTCR_EL2 register is done >> out of the cpu_up/start_secondary control flow. However, the cpu_up flow >> should be self-contained - it should fully initialize a secondary CPU, >> because the cpu_up is used not only to bring a secondary CPU online on >> boot, but also to hotplug a CPU during the system resume. >> With this patch the setting of paging is triggered from start_secondary >> function if the current system state is not boot. This way, the paging >> will be setup in non-boot scenarios, while the setup in boot scenario >> remains unchanged. > > > I am afraid that this is not correct. You can't assume that value chosen for > VTCR by Xen at boot will fit this new CPU. So you have to check it is fine > or park the CPU if there are any issue. > This is not a new CPU. This CPU already went through its boot sequence and it reached the resume point because it does fit the value chosen for VTCR by Xen. If it wouldn't fit the chosen value for VTCR it would be parked so it wouldn't participate in suspend/resume. Please let me know if I misunderstood your comment. AFAIU the value chosen by Xen for VTCR config has to be common for all online CPUs. Since this value is also used in the resume path I suggest to make global (static in the p2m.c) the 'val' variable which is currently local in setup_virt_paging() and passed as argument to setup_virt_paging_one(). Then setup_virt_paging_one() would not receive an argument. I need to access this value on resume, so I would call setup_virt_paging_one() without argument from start_secondary() if the system state is not boot. This seems to me a bit cleaner compared to what I submitted in this patch, but fundamentally the functionality is the same. Thanks, Mirela > For more details have a look at [1]. > > [1] > https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg02482.html > > Cheers, > > -- > Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH 6/7] xen/arm: Setup virtual paging for secondary CPUs in non-boot scenario
Hi, On 11/04/18 14:19, Mirela Simonovic wrote: In existing code the paging for secondary CPUs is setup only in boot flow. The setup is triggered from start_xen function after all CPUs are brought online. In other words, the initialization of VTCR_EL2 register is done out of the cpu_up/start_secondary control flow. However, the cpu_up flow should be self-contained - it should fully initialize a secondary CPU, because the cpu_up is used not only to bring a secondary CPU online on boot, but also to hotplug a CPU during the system resume. With this patch the setting of paging is triggered from start_secondary function if the current system state is not boot. This way, the paging will be setup in non-boot scenarios, while the setup in boot scenario remains unchanged. I am afraid that this is not correct. You can't assume that value chosen for VTCR by Xen at boot will fit this new CPU. So you have to check it is fine or park the CPU if there are any issue. For more details have a look at [1]. [1] https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg02482.html Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
[Xen-devel] [PATCH 6/7] xen/arm: Setup virtual paging for secondary CPUs in non-boot scenario
In existing code the paging for secondary CPUs is setup only in boot flow. The setup is triggered from start_xen function after all CPUs are brought online. In other words, the initialization of VTCR_EL2 register is done out of the cpu_up/start_secondary control flow. However, the cpu_up flow should be self-contained - it should fully initialize a secondary CPU, because the cpu_up is used not only to bring a secondary CPU online on boot, but also to hotplug a CPU during the system resume. With this patch the setting of paging is triggered from start_secondary function if the current system state is not boot. This way, the paging will be setup in non-boot scenarios, while the setup in boot scenario remains unchanged. Signed-off-by: Mirela Simonovic--- xen/arch/arm/p2m.c| 10 -- xen/arch/arm/smpboot.c| 3 +++ xen/include/asm-arm/p2m.h | 3 +++ 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c index d43c3aa896..d041b4d24b 100644 --- a/xen/arch/arm/p2m.c +++ b/xen/arch/arm/p2m.c @@ -1451,9 +1451,15 @@ err: return page; } -static void __init setup_virt_paging_one(void *data) +void setup_virt_paging_one(void *data) { -unsigned long val = (unsigned long)data; +static unsigned long vtcr_data; +unsigned long val; + +if ( data ) +vtcr_data = (unsigned long)data; +val = vtcr_data; + WRITE_SYSREG32(val, VTCR_EL2); isb(); } diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c index d15ea8df5e..dae4427320 100644 --- a/xen/arch/arm/smpboot.c +++ b/xen/arch/arm/smpboot.c @@ -360,6 +360,9 @@ void start_secondary(unsigned long boot_phys_offset, check_local_cpu_errata(); +if ( system_state != SYS_STATE_boot ) +setup_virt_paging_one(NULL); + printk(XENLOG_DEBUG "CPU %u booted.\n", smp_processor_id()); startup_cpu_idle_loop(); diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h index 8823707c17..6cc36ebbc3 100644 --- a/xen/include/asm-arm/p2m.h +++ b/xen/include/asm-arm/p2m.h @@ -153,6 +153,9 @@ void p2m_altp2m_check(struct vcpu *v, uint16_t idx) /* Second stage paging setup, to be called on all CPUs */ void setup_virt_paging(void); +/* Second stage paging setup, to be called by one CPU out of the boot flow */ +void setup_virt_paging_one(void *data); + /* Init the datastructures for later use by the p2m code */ int p2m_init(struct domain *d); -- 2.13.0 ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel