Re: [PATCH v5 2/2] x86/xen: allow privcmd hypercalls to be preempted on 64-bit
On 03/02/15 00:24, Luis R. Rodriguez wrote: > > As I waited I decided to finally install a 32-bit OS but I found now > that as of Xen commit 5d1181a5 which went upstream as of xen 4.3 Xen > no longer supports 32-bit for x86. You don't need a 32-bit hypervisor to run 32-bit dom0 or domUs. David -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 2/2] x86/xen: allow privcmd hypercalls to be preempted on 64-bit
On Thu, Jan 29, 2015 at 12:35 PM, Luis R. Rodriguez wrote: > On Tue, Jan 27, 2015 at 10:06:44AM +, David Vrabel wrote: >> On 27/01/15 08:35, Jan Beulich wrote: >> On 27.01.15 at 02:51, wrote: >> > >> > Even if David told you this would be acceptable, I have to question >> > an abstract model of fixing issues on only 64-bit kernels - this may >> > be acceptable for distro purposes, but seems hardly the right >> > approach for upstream. If 32-bit ones are to become deliberately >> > broken, the XEN config option should become dependent on !X86_32. >> >> I'd rather have something omitted (keeping the current behaviour) than >> something that has not been tested at all. >> >> Obviously it would be preferable to to fix both 32-bit and 64-bit x86 >> (and ARM as well) but I'm not going to block an important bug fix for >> the majority use case (64-bit x86). > > Hey folks, what is the status of these patches? Any more feedback? As I waited I decided to finally install a 32-bit OS but I found now that as of Xen commit 5d1181a5 which went upstream as of xen 4.3 Xen no longer supports 32-bit for x86. So -- Xen no loner even builds on 32-bit systems. The latest version I could use then would be Xen 4.2 but the even the stable-4.2 branch fails to build for me even after fixing quite a bit of compile failures, I'm just going to give up now. I want to build Xen as I'm relying on some customized emulated delays on the hypervisor in order to test the issue, otherwise I'd have to dedicate a huge system to reproduce this issue. While it is possible folks on old xen may use newer kernels this issue is likely not a high priority on those 32-bit systems and likely hard to reproduce there. When and if someone is able to test this on a 32-bit kernel / hypervisor then they can test the small patch below. Can we move forward with the 64-bit part then? diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S index 000d419..b4b1f42 100644 --- a/arch/x86/kernel/entry_32.S +++ b/arch/x86/kernel/entry_32.S @@ -982,6 +982,8 @@ ENTRY(xen_hypervisor_callback) ENTRY(xen_do_upcall) 1:mov %esp, %eax call xen_evtchn_do_upcall +movl %esp,%eax +call xen_end_upcall jmp ret_from_intr CFI_ENDPROC ENDPROC(xen_hypervisor_callback) -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 2/2] x86/xen: allow privcmd hypercalls to be preempted on 64-bit
On Tue, Jan 27, 2015 at 10:06:44AM +, David Vrabel wrote: > On 27/01/15 08:35, Jan Beulich wrote: > On 27.01.15 at 02:51, wrote: > > > > Even if David told you this would be acceptable, I have to question > > an abstract model of fixing issues on only 64-bit kernels - this may > > be acceptable for distro purposes, but seems hardly the right > > approach for upstream. If 32-bit ones are to become deliberately > > broken, the XEN config option should become dependent on !X86_32. > > I'd rather have something omitted (keeping the current behaviour) than > something that has not been tested at all. > > Obviously it would be preferable to to fix both 32-bit and 64-bit x86 > (and ARM as well) but I'm not going to block an important bug fix for > the majority use case (64-bit x86). Hey folks, what is the status of these patches? Any more feedback? Luis -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 2/2] x86/xen: allow privcmd hypercalls to be preempted on 64-bit
On Tue, Jan 27, 2015 at 10:06:44AM +, David Vrabel wrote: > On 27/01/15 08:35, Jan Beulich wrote: > On 27.01.15 at 02:51, wrote: > > > > Even if David told you this would be acceptable, I have to question > > an abstract model of fixing issues on only 64-bit kernels - this may > > be acceptable for distro purposes, but seems hardly the right > > approach for upstream. If 32-bit ones are to become deliberately > > broken, the XEN config option should become dependent on !X86_32. > > I'd rather have something omitted (keeping the current behaviour) than > something that has not been tested at all. > > Obviously it would be preferable to to fix both 32-bit and 64-bit x86 > (and ARM as well) but I'm not going to block an important bug fix for > the majority use case (64-bit x86). The hunk for 32-bit should indeed only go in once we get a Tested-by. Please let me know if there is anything else needed for this patch. Luis -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 2/2] x86/xen: allow privcmd hypercalls to be preempted on 64-bit
On 27/01/15 08:35, Jan Beulich wrote: On 27.01.15 at 02:51, wrote: > > Even if David told you this would be acceptable, I have to question > an abstract model of fixing issues on only 64-bit kernels - this may > be acceptable for distro purposes, but seems hardly the right > approach for upstream. If 32-bit ones are to become deliberately > broken, the XEN config option should become dependent on !X86_32. I'd rather have something omitted (keeping the current behaviour) than something that has not been tested at all. Obviously it would be preferable to to fix both 32-bit and 64-bit x86 (and ARM as well) but I'm not going to block an important bug fix for the majority use case (64-bit x86). David -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Xen-devel] [PATCH v5 2/2] x86/xen: allow privcmd hypercalls to be preempted on 64-bit
On 27/01/15 08:35, Jan Beulich wrote: On 27.01.15 at 02:51, wrote: > Even if David told you this would be acceptable, I have to question > an abstract model of fixing issues on only 64-bit kernels - this may > be acceptable for distro purposes, but seems hardly the right > approach for upstream. If 32-bit ones are to become deliberately > broken, the XEN config option should become dependent on !X86_32. There are still legitimate reasons to prefer 32bit PV guests over 64bit ones. Previous versions of this patch had 32bit support as well. Why did you drop it? ~Andrew -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 2/2] x86/xen: allow privcmd hypercalls to be preempted on 64-bit
>>> On 27.01.15 at 02:51, wrote: Even if David told you this would be acceptable, I have to question an abstract model of fixing issues on only 64-bit kernels - this may be acceptable for distro purposes, but seems hardly the right approach for upstream. If 32-bit ones are to become deliberately broken, the XEN config option should become dependent on !X86_32. Jan -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v5 2/2] x86/xen: allow privcmd hypercalls to be preempted on 64-bit
From: "Luis R. Rodriguez"
Xen has support for splitting heavy work work into a series
of hypercalls, called multicalls, and preempting them through
what Xen calls continuation [0]. Despite this though without
CONFIG_PREEMPT preemption won't happen, without preemption
a system can become pretty useless on heavy handed hypercalls.
Such is the case for example when creating a > 50 GiB HVM guest,
we can get softlockups [1] with:.
kernel: [ 802.084335] BUG: soft lockup - CPU#1 stuck for 22s! [xend:31351]
The softlock up triggers on the TASK_UNINTERRUPTIBLE hanger check
(default 120 seconds), on the Xen side in this particular case
this happens when the following Xen hypervisor code is used:
xc_domain_set_pod_target() -->
do_memory_op() -->
arch_memory_op() -->
p2m_pod_set_mem_target()
-- long delay (real or emulated) --
This happens on arch_memory_op() on the XENMEM_set_pod_target memory
op even though arch_memory_op() can handle continuation via
hypercall_create_continuation() for example.
Machines over 50 GiB of memory are on high demand and hard to come
by so to help replicate this sort of issue long delays on select
hypercalls have been emulated in order to be able to test this on
smaller machines [2].
On one hand this issue can be considered as expected given that
CONFIG_PREEMPT=n is used however we have forced voluntary preemption
precedent practices in the kernel even for CONFIG_PREEMPT=n through
the usage of cond_resched() sprinkled in many places. To address
this issue with Xen hypercalls though we need to find a way to aid
to the schedular in the middle of hypercalls. We are motivated to
address this issue on CONFIG_PREEMPT=n as otherwise the system becomes
rather unresponsive for long periods of time; in the worst case, at least
only currently by emulating long delays on select io disk bound
hypercalls, this can lead to filesystem corruption if the delay happens
for example on SCHEDOP_remote_shutdown (when we call 'xl shutdown').
We can address this problem by trying to check if we should schedule
on the xen timer in the middle of a hypercall on the return from the
timer interrupt. We want to be careful to not always force voluntary
preemption though so to do this we only selectively enable preemption
on very specific xen hypercalls.
This enables hypercall preemption by selectively forcing checks for
voluntary preempting only on ioctl initiated private hypercalls
where we know some folks have run into reported issues [1].
[0]
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=42217cbc5b3e84b8c145d8cfb62dd5de0134b9e8;hp=3a0b9c57d5c9e82c55dd967c84dd06cb43c49ee9
[1] https://bugzilla.novell.com/show_bug.cgi?id=861093
[2] http://ftp.suse.com/pub/people/mcgrof/xen/emulate-long-xen-hypercalls.patch
Based on original work by: David Vrabel
Suggested-by: Andy Lutomirski
Cc: Andy Lutomirski
Cc: Borislav Petkov
Cc: David Vrabel
Cc: Thomas Gleixner
Cc: Ingo Molnar
Cc: "H. Peter Anvin"
Cc: x...@kernel.org
Cc: Steven Rostedt
Cc: Masami Hiramatsu
Cc: Jan Beulich
Cc: linux-ker...@vger.kernel.org
Reviewed-by: Andy Lutomirski
Signed-off-by: Luis R. Rodriguez
---
arch/x86/kernel/entry_64.S | 2 ++
drivers/xen/events/events_base.c | 14 ++
include/xen/events.h | 1 +
3 files changed, 17 insertions(+)
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index 9ebaf63..ee28733 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -1198,6 +1198,8 @@ ENTRY(xen_do_hypervisor_callback) #
do_hypervisor_callback(struct *pt_regs)
popq %rsp
CFI_DEF_CFA_REGISTER rsp
decl PER_CPU_VAR(irq_count)
+ movq %rsp, %rdi /* pass pt_regs as first argument */
+ call xen_end_upcall
jmp error_exit
CFI_ENDPROC
END(xen_do_hypervisor_callback)
diff --git a/drivers/xen/events/events_base.c b/drivers/xen/events/events_base.c
index b4bca2d..bf207f2 100644
--- a/drivers/xen/events/events_base.c
+++ b/drivers/xen/events/events_base.c
@@ -32,6 +32,8 @@
#include
#include
#include
+#include
+#include
#ifdef CONFIG_X86
#include
@@ -1243,6 +1245,18 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)
set_irq_regs(old_regs);
}
+/*
+ * Some hypercalls issued by the toolstack can take many 10s of
+ * seconds. Allow tasks running hypercalls via the privcmd driver to be
+ * voluntarily preempted even if full kernel preemption is disabled.
+ */
+void xen_end_upcall(struct pt_regs *regs)
+{
+ if (xen_is_preemptible_hypercall(regs))
+ _cond_resched();
+}
+NOKPROBE_SYMBOL(xen_end_upcall);
+
void xen_hvm_evtchn_do_upcall(void)
{
__xen_evtchn_do_upcall();
diff --git a/include/xen/events.h b/include/xen/events.h
index 5321cd9..f08df87 100644
--- a/include/xen/events.h
+++ b/include/xen/events.h
@@ -95,6 +95,7 @@ void xen_hvm_callback_vector(void);
extern int xen_have_vector_callback;
int xen_set_callback_via(uint64_t via);
void xen_evtc
