[Qemu-devel] livepatch kernel to use md-clear without reboot

[Stefan Priebe - Profihost AG]

Hello,

i'm interested if it is possible to livepatch the guest kernel without
reboot.

(host kernel and microcode is patched)

Greets,
Stefan

Re: [Qemu-devel] cpu.fail / MDS fixes

[Stefan Priebe - Profihost AG]

Am 15.05.19 um 19:54 schrieb Daniel P. Berrangé:
> On Wed, May 15, 2019 at 07:13:56PM +0200, Stefan Priebe - Profihost AG wrote:
>> Hello list,
>>
>> i've updated my host to kernel 4.19.43 and applied the following patch
>> to my qemu 2.12.1:
>> https://bugzilla.suse.com/attachment.cgi?id=798722
>>
>> But my guest running 4.19.43 still says:
>> Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state
>> unknown
>>
>> while the host says:
>> Vulnerable: Clear CPU buffers attempted, SMT Host state unknown
> 
> That suggests your host OS hasn't got the new microcode installed
> or has not loaded it.

No it does not. A not loaded Microcode looks like this:
Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable

but in my case it is:
Mitigation: Clear CPU buffers; SMT vulnerable

on the host as hyper threading is still enabled.

> You want the host to report that it is Mitigated, and for the
> host's /proc/cpuinfo to report "md-clear" exists.
> 
>> I expected the guest can use the new microcode.
> 
> You've not said what CPU model you've given to the guest.
> 
> You need either "-cpu host", or if using a named CPU model
> you need to explicitly turn on the "md-clear" feature
> (and all previous fixes)
> 
>eg  "-cpu Haswell,+spec-ctrl,+ssbd,+md-clear"
hah yes you're true i need to specifiy +md-clean

Thanks!

> Regards,
> Daniel
> 

[Qemu-devel] cpu.fail / MDS fixes

[Stefan Priebe - Profihost AG]

Hello list,

i've updated my host to kernel 4.19.43 and applied the following patch
to my qemu 2.12.1:
https://bugzilla.suse.com/attachment.cgi?id=798722

But my guest running 4.19.43 still says:
Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state
unknown

while the host says:
Vulnerable: Clear CPU buffers attempted, SMT Host state unknown

I expected the guest can use the new microcode.

Greets,
Stefan

Re: [Qemu-devel] Overcommiting cpu results in all vms offline

[Stefan Priebe - Profihost AG]

Am 17.09.2018 um 11:40 schrieb Jack Wang:
> Stefan Priebe - Profihost AG  于2018年9月17日周一 上午9:00写道：
>>
>> Hi,
>>
>> Am 17.09.2018 um 08:38 schrieb Jack Wang:
>>> Stefan Priebe - Profihost AG  于2018年9月16日周日 下午3:31写道：
>>>>
>>>> Hello,
>>>>
>>>> while overcommiting cpu I had several situations where all vms gone 
>>>> offline while two vms saturated all cores.
>>>>
>>>> I believed all vms would stay online but would just not be able to use all 
>>>> their cores?
>>>>
>>>> My original idea was to automate live migration on high host load to move 
>>>> vms to another node but that makes only sense if all vms stay online.
>>>>
>>>> Is this expected? Anything special needed to archive this?
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>> Hi, Stefan,
>>>
>>> Do you have any logs when all VMs go offline?
>>> Maybe OOMkiller play a role there?
>>
>> After reviewing i think this is memory related but OOM did not play a role.
>> All kvm processes where spinning trying to get > 100% CPU and i was not
>> able to even login to ssh. After 5-10 minutes i was able to login.
> So the VMs are not really offline, what the result if you run
> query-status via qmp?

I can't as i can't connect to the host in that stage.

>> There were about 150GB free mem.
>>
>> Relevant settings (no local storage involved):
>> vm.dirty_background_ratio:
>> 3
>> vm.dirty_ratio:
>> 10
>> vm.min_free_kbytes:
>> 10567004
>>
>> # cat /sys/kernel/mm/transparent_hugepage/defrag
>> always defer [defer+madvise] madvise never
>>
>> # cat /sys/kernel/mm/transparent_hugepage/enabled
>> [always] madvise never
>>
>> After that i had the following traces on the host node:
>> https://pastebin.com/raw/0VhyQmAv
> 
> The call trace looks ceph related deadlock or hung.

Yes but i can also show you traces where nothing from ceph is involved
the only thing they have in common is the beginning in page_fault.

>> Thanks!
>>
>> Greets,
>> Stefan

Re: [Qemu-devel] Overcommiting cpu results in all vms offline

[Stefan Priebe - Profihost AG]

May be amissing piece:
vm.overcommit_memory=0

Greets,
Stefan


Am 17.09.2018 um 09:00 schrieb Stefan Priebe - Profihost AG:
> Hi,
> 
> Am 17.09.2018 um 08:38 schrieb Jack Wang:
>> Stefan Priebe - Profihost AG  于2018年9月16日周日 下午3:31写道：
>>>
>>> Hello,
>>>
>>> while overcommiting cpu I had several situations where all vms gone offline 
>>> while two vms saturated all cores.
>>>
>>> I believed all vms would stay online but would just not be able to use all 
>>> their cores?
>>>
>>> My original idea was to automate live migration on high host load to move 
>>> vms to another node but that makes only sense if all vms stay online.
>>>
>>> Is this expected? Anything special needed to archive this?
>>>
>>> Greets,
>>> Stefan
>>>
>> Hi, Stefan,
>>
>> Do you have any logs when all VMs go offline?
>> Maybe OOMkiller play a role there?
> 
> After reviewing i think this is memory related but OOM did not play a role.
> All kvm processes where spinning trying to get > 100% CPU and i was not
> able to even login to ssh. After 5-10 minutes i was able to login.
> 
> There were about 150GB free mem.
> 
> Relevant settings (no local storage involved):
> vm.dirty_background_ratio:
> 3
> vm.dirty_ratio:
> 10
> vm.min_free_kbytes:
> 10567004
> 
> # cat /sys/kernel/mm/transparent_hugepage/defrag
> always defer [defer+madvise] madvise never
> 
> # cat /sys/kernel/mm/transparent_hugepage/enabled
> [always] madvise never
> 
> After that i had the following traces on the host node:
> https://pastebin.com/raw/0VhyQmAv
> 
> Thanks!
> 
> Greets,
> Stefan
> 

Re: [Qemu-devel] Overcommiting cpu results in all vms offline

[Stefan Priebe - Profihost AG]

Hi,

Am 17.09.2018 um 08:38 schrieb Jack Wang:
> Stefan Priebe - Profihost AG  于2018年9月16日周日 下午3:31写道：
>>
>> Hello,
>>
>> while overcommiting cpu I had several situations where all vms gone offline 
>> while two vms saturated all cores.
>>
>> I believed all vms would stay online but would just not be able to use all 
>> their cores?
>>
>> My original idea was to automate live migration on high host load to move 
>> vms to another node but that makes only sense if all vms stay online.
>>
>> Is this expected? Anything special needed to archive this?
>>
>> Greets,
>> Stefan
>>
> Hi, Stefan,
> 
> Do you have any logs when all VMs go offline?
> Maybe OOMkiller play a role there?

After reviewing i think this is memory related but OOM did not play a role.
All kvm processes where spinning trying to get > 100% CPU and i was not
able to even login to ssh. After 5-10 minutes i was able to login.

There were about 150GB free mem.

Relevant settings (no local storage involved):
vm.dirty_background_ratio:
3
vm.dirty_ratio:
10
vm.min_free_kbytes:
10567004

# cat /sys/kernel/mm/transparent_hugepage/defrag
always defer [defer+madvise] madvise never

# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

After that i had the following traces on the host node:
https://pastebin.com/raw/0VhyQmAv

Thanks!

Greets,
Stefan

[Qemu-devel] Overcommiting cpu results in all vms offline

[Stefan Priebe - Profihost AG]

Hello,

while overcommiting cpu I had several situations where all vms gone offline 
while two vms saturated all cores.

I believed all vms would stay online but would just not be able to use all 
their cores?

My original idea was to automate live migration on high host load to move vms 
to another node but that makes only sense if all vms stay online.

Is this expected? Anything special needed to archive this?

Greets,
Stefan

Re: [Qemu-devel] Qemu and Spectre_V4 + l1tf + IBRS_FW

[Stefan Priebe - Profihost AG]

Am 17.08.2018 um 11:41 schrieb Daniel P. Berrangé:
> On Fri, Aug 17, 2018 at 08:44:38AM +0200, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> i haven't found anything on the web regarding qemu and mentioned variants.
>>
>> While my host says:
>> l1tf:Mitigation: PTE Inversion; VMX: SMT vulnerable, L1D conditional
>> cache flushes
>> meltdown:Mitigation: PTI
>> spec_store_bypass:Mitigation: Speculative Store Bypass disabled via
>> prctl and seccomp
>> spectre_v1:Mitigation: __user pointer sanitization
>> spectre_v2:Mitigation: Full generic retpoline, IBPB, IBRS_FW
>>
>> My guests bootet with pcid and spec-ctrl only say:
>> l1tf:Mitigation: PTE Inversion
>> meltdown:Mitigation: PTI
>> spec_store_bypass:Vulnerable
>> spectre_v1:Mitigation: __user pointer sanitization
>> spectre_v2:Mitigation: Full generic retpoline, IBPB
>>
>> * What is about spec_store_bypass in Qemu?
> 
> The guest needs an 'ssbd' feature for Intel CPU models and either a
> 'virt-ssbd' or 'amd-ssbd' feature for AMD CPU models.

Ah thanks. That works fine.

>> * What is about IBRS_FW feature?
> 
> I'm not sure what IBRS_FW is referring to, but don't worry about it.
> The fact the the guest kernel says "Mitigation" instead of "Vulnerable"
> means you are protected with your current config.
> 
> For Intel CPU models Spectre v2 needs the guest to have the 'spec-ctrl'
> feature. On AMD models Spectre v2 the guest needs 'ibpb' feature.
> 
>> * What is about L1TF?
> 
> No extra CPU flags are required for QEMU guests for L1TF. The new CPU
> feature is merely an perf optimization for the host hypervisor fixes.
> 
> Note that with L1TF there are extra steps you need to consider wrt
> hyperthreading, that won't be reflected in the 'vulnerabilities'
> data published by the kernel.
> 
> You can read more about the procedure for dealing with L1TF in
> virt hosts in the "Resolve" tab of this article:
> 
>   https://access.redhat.com/security/vulnerabilities/L1TF
> 
>> Or are those just irrelevant to Qemu guests? Would be great to have some
>> informations.
> 
> We have some QEMU docs providing guidance on guest CPU model/feature config
> but they are not yet published. In the meantime this blog post of mine gives
> the same info, covering what's needed for Spectre v2, Meltdown and SSBD and
> guidance in general for CPU config:
> 
>   
> https://www.berrange.com/posts/2018/06/29/cpu-model-configuration-for-qemu-kvm-on-x86-hosts/

Thanks, that one was really helpful.

Greets,
Stefan

[Qemu-devel] Qemu and Spectre_V4 + l1tf + IBRS_FW

[Stefan Priebe - Profihost AG]

Hello,

i haven't found anything on the web regarding qemu and mentioned variants.

While my host says:
l1tf:Mitigation: PTE Inversion; VMX: SMT vulnerable, L1D conditional
cache flushes
meltdown:Mitigation: PTI
spec_store_bypass:Mitigation: Speculative Store Bypass disabled via
prctl and seccomp
spectre_v1:Mitigation: __user pointer sanitization
spectre_v2:Mitigation: Full generic retpoline, IBPB, IBRS_FW

My guests bootet with pcid and spec-ctrl only say:
l1tf:Mitigation: PTE Inversion
meltdown:Mitigation: PTI
spec_store_bypass:Vulnerable
spectre_v1:Mitigation: __user pointer sanitization
spectre_v2:Mitigation: Full generic retpoline, IBPB

* What is about spec_store_bypass in Qemu?
* What is about IBRS_FW feature?
* What is about L1TF?

Or are those just irrelevant to Qemu guests? Would be great to have some
informations.

Thanks a lot!

Greets,
Stefan

Re: [Qemu-devel] how to pass pcid to guest?

[Stefan Priebe - Profihost AG]

Am 08.01.2018 um 23:07 schrieb Eric Blake:
> On 01/08/2018 02:03 PM, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> for meltdown mitigation and performance it's important to have the pcid
>> flag passed down to the guest (f.e.
>> https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU).
> 
> Indeed; you are still waiting on the qemu patch mentioned here:
> https://www.qemu.org/2018/01/04/spectre/
> 
> which is still undergoing the review process, but should be up (in the
> form of 2.11.1) "in the next few days".

OK thanks as the performance difference is significant:

no pcid:
# time for i in $(seq 1 1 50); do du -sx /; done
...
real0m26.614s
user0m17.548s
sys 0m9.056s


kvm started with +pcid:
# time for i in $(seq 1 1 50); do du -sx /; done
...
real0m14.734s
user0m7.755s
sys 0m6.973s

Greets,
Stefan

[Qemu-devel] how to pass pcid to guest?

[Stefan Priebe - Profihost AG]

Hello,

for meltdown mitigation and performance it's important to have the pcid
flag passed down to the guest (f.e.
https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU).

My host shows the flag:
# grep ' pcid ' /proc/cpuinfo  | wc -l
56

But the guest does not:
# grep pcid /proc/cpuinfo
#

Guest was started with:
-cpu IvyBridge,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,vendor=GenuineIntel

Qemu is 2.9.1

Thanks!

Greets,
Stefan

[Qemu-devel] What's needed to pass pciid cpu flag to guest?

[Stefan Priebe - Profihost AG]

Hello,

for meltdown mitigation and performance it's important to have the pcid
flag passed down to the guest (f.e.
https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU).

My host shows the flag:
# grep ' pcid ' /proc/cpuinfo  | wc -l
56

But the guest does not:
# grep pcid /proc/cpuinfo
#

Guest was started with:
-cpu IvyBridge,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,vendor=GenuineIntel

Qemu is 2.9.1

Thanks!

Greets,
Stefan

Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches

[Stefan Priebe - Profihost AG]

Thanks! But it‘s very difficult to get all opinions all together.

SuSE Enterprise told me to update:
- kernel
- qemu
- Intel microcode

And the released already updates for all of them.

Stefan

Excuse my typo sent from my mobile phone.

> Am 05.01.2018 um 09:33 schrieb Paolo Bonzini <pbonz...@redhat.com>:
> 
>> On 04/01/2018 21:15, Stefan Priebe - Profihost AG wrote:
>> attached the relevant patch for everybody who needs it.
> 
> This is the original patch from Intel, which doesn't work unless you
> have a patched kernel (which you almost certainly don't have) and
> doesn't even warn you about that.
> 
> In other words, it's rubbish.  Please read
> https://www.qemu.org/2018/01/04/spectre/ several times, until you
> understand why there is no urgent need to update QEMU.
> 
> Days are 24 hours for QEMU developers just like for you (and believe me,
> we wished several times that they weren't during the last two months).
> We are prioritizing the fixes according to their effect in mitigating
> the vulnerability, their applicability and the availability of patches
> to the lower levels of the stack.  Right now, the most urgent part is
> the simple mitigations that can go in Linux 4.15 and stable kernels.
> 
> Paolo

Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches

[Stefan Priebe - Profihost AG]

attached the relevant patch for everybody who needs it.

Greets,
Stefan
Am 04.01.2018 um 16:53 schrieb Paolo Bonzini:
> On 04/01/2018 09:35, Alexandre DERUMIER wrote:
>>>> So you need: 
>>>> 1.) intel / amd cpu microcode update 
>>>> 2.) qemu update to pass the new MSR and CPU flags from the microcode 
>>>> update 
>>>> 3.) host kernel update 
>>>> 4.) guest kernel update 
>>
>> are you sure we need to patch guest kernel if we are able to patch qemu ?
> 
> Patching the guest kernel is only required to protect the guest kernel
> from guest usermode.
> 
>> If I understand, patching the host kernel, should avoid that a vm is reading 
>> memory of another vm.
>> (the most critical)
> 
> Correct.
> 
>> patching the guest kernel, to avoid that a process from the vm have access 
>> to memory of another process of same vm.
> 
> Correct.
> 
> The QEMU updates are pretty boring, mostly taking care of new MSR and
> CPUID flags (and adding new CPU models).
> 
> They are not needed to protect the guest from "Meltdown", only
> "Spectre"---the former only needs a guest kernel update.  Also, to have
> any effect, the guest kernels must also have "Spectre" patches which
> aren't upstream yet for either KVM or the rest of Linux.  So the QEMU
> patches are much less important than the kernel side.
> 
>>> https://access.redhat.com/solutions/3307851 
>>> "Impacts of CVE-2017-5754, CVE-2017-5753, and CVE-2017-5715 to Red Hat 
>>> Virtualization products" 
> 
> It mostly repeats the contents of the RHEL document
> https://access.redhat.com/security/vulnerabilities/speculativeexecution,
> with some information specific to RHV.
> 
> Thanks,
> 
> Paolo
> 
>> i don't have one but the content might be something like this: 
>> https://www.suse.com/de-de/support/kb/doc/?id=7022512 
>>
>> So you need: 
>> 1.) intel / amd cpu microcode update 
>> 2.) qemu update to pass the new MSR and CPU flags from the microcode update 
>> 3.) host kernel update 
>> 4.) guest kernel update 
>>
>> The microcode update and the kernel update is publicly available but i'm 
>> missing the qemu one. 
>>
>> Greets, 
>> Stefan 
>>
>>> - Mail original - 
>>> De: "aderumier" <aderum...@odiso.com> 
>>> À: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag> 
>>> Cc: "qemu-devel" <qemu-devel@nongnu.org> 
>>> Envoyé: Jeudi 4 Janvier 2018 08:24:34 
>>> Objet: Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches 
>>>
>>>>> Can anybody point me to the relevant qemu patches? 
>>>
>>> I don't have find them yet. 
>>>
>>> Do you known if a vm using kvm64 cpu model is protected or not ? 
>>>
>>> - Mail original - 
>>> De: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag> 
>>> À: "qemu-devel" <qemu-devel@nongnu.org> 
>>> Envoyé: Jeudi 4 Janvier 2018 07:27:01 
>>> Objet: [Qemu-devel] CVE-2017-5715: relevant qemu patches 
>>>
>>> Hello, 
>>>
>>> i've seen some vendors have updated qemu regarding meltdown / spectre. 
>>>
>>> f.e.: 
>>>
>>> CVE-2017-5715: QEMU was updated to allow passing through new MSR and 
>>> CPUID flags from the host VM to the CPU, to allow enabling/disabling 
>>> branch prediction features in the Intel CPU. (bsc#1068032) 
>>>
>>> Can anybody point me to the relevant qemu patches? 
>>>
>>> Thanks! 
>>>
>>> Greets, 
>>> Stefan 
>>>
>>
>>
> 
>From b4fdfeb4545c09a0fdf01edc938f9cce8fcaa5c6 Mon Sep 17 00:00:00 2001
From: Wei Wang <wei.w.w...@intel.com>
Date: Tue, 7 Nov 2017 16:39:49 +0800
Subject: [PATCH] i386/kvm: MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD

CPUID(EAX=0X7,ECX=0).EDX[26]/[27] indicates the support of
MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD. Expose the CPUID
to the guest. Also add the support of transferring the MSRs during live
migration.

Signed-off-by: Wei Wang <wei.w.w...@intel.com>
[BR: BSC#1068032 CVE-2017-5715]
Signed-off-by: Bruce Rogers <brog...@suse.com>
---
 target/i386/cpu.c |  3 ++-
 target/i386/cpu.h |  4 
 target/i386/kvm.c | 15 ++-
 target/i386/machine.c | 20 
 4 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 55f72b679f..01761db3fc 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -2823,13 +2823,14 @@ vo

Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches

[Stefan Priebe - Profihost AG]

Nobody? Is this something they did on their own?

Stefan

Am 04.01.2018 um 07:27 schrieb Stefan Priebe - Profihost AG:
> Hello,
> 
> i've seen some vendors have updated qemu regarding meltdown / spectre.
> 
> f.e.:
> 
>  CVE-2017-5715: QEMU was updated to allow passing through new MSR and
>  CPUID flags from the host VM to the CPU, to allow enabling/disabling
>  branch prediction features in the Intel CPU. (bsc#1068032)
> 
> Can anybody point me to the relevant qemu patches?
> 
> Thanks!
> 
> Greets,
> Stefan
> 

Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches

[Stefan Priebe - Profihost AG]

Am 04.01.2018 um 09:35 schrieb Alexandre DERUMIER:
>>> So you need: 
>>> 1.) intel / amd cpu microcode update 
>>> 2.) qemu update to pass the new MSR and CPU flags from the microcode update 
>>> 3.) host kernel update 
>>> 4.) guest kernel update 
> 
> are you sure we need to patch guest kernel if we are able to patch qemu ?
>> I have some pretty old guest (linux and windows)
>
> If I understand, patching the host kernel, should avoid that a vm is reading 
> memory of another vm.
> (the most critical)

Yes - this was just to complete the mitigation on all layers.

> 
> patching the guest kernel, to avoid that a process from the vm have access to 
> memory of another process of same vm.
Yes.

Stefan

> 
> 
> 
> - Mail original -
> De: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag>
> À: "aderumier" <aderum...@odiso.com>
> Cc: "qemu-devel" <qemu-devel@nongnu.org>
> Envoyé: Jeudi 4 Janvier 2018 09:17:41
> Objet: Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches
> 
> Am 04.01.2018 um 08:27 schrieb Alexandre DERUMIER: 
>> does somebody have a redhat account to see te content of: 
>>
>> https://access.redhat.com/solutions/3307851 
>> "Impacts of CVE-2017-5754, CVE-2017-5753, and CVE-2017-5715 to Red Hat 
>> Virtualization products" 
> 
> i don't have one but the content might be something like this: 
> https://www.suse.com/de-de/support/kb/doc/?id=7022512 
> 
> So you need: 
> 1.) intel / amd cpu microcode update 
> 2.) qemu update to pass the new MSR and CPU flags from the microcode update 
> 3.) host kernel update 
> 4.) guest kernel update 
> 
> The microcode update and the kernel update is publicly available but i'm 
> missing the qemu one. 
> 
> Greets, 
> Stefan 
> 
>> - Mail original - 
>> De: "aderumier" <aderum...@odiso.com> 
>> À: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag> 
>> Cc: "qemu-devel" <qemu-devel@nongnu.org> 
>> Envoyé: Jeudi 4 Janvier 2018 08:24:34 
>> Objet: Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches 
>>
>>>> Can anybody point me to the relevant qemu patches? 
>>
>> I don't have find them yet. 
>>
>> Do you known if a vm using kvm64 cpu model is protected or not ? 
>>
>> - Mail original - 
>> De: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag> 
>> À: "qemu-devel" <qemu-devel@nongnu.org> 
>> Envoyé: Jeudi 4 Janvier 2018 07:27:01 
>> Objet: [Qemu-devel] CVE-2017-5715: relevant qemu patches 
>>
>> Hello, 
>>
>> i've seen some vendors have updated qemu regarding meltdown / spectre. 
>>
>> f.e.: 
>>
>> CVE-2017-5715: QEMU was updated to allow passing through new MSR and 
>> CPUID flags from the host VM to the CPU, to allow enabling/disabling 
>> branch prediction features in the Intel CPU. (bsc#1068032) 
>>
>> Can anybody point me to the relevant qemu patches? 
>>
>> Thanks! 
>>
>> Greets, 
>> Stefan 
>>
> 

Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches

[Stefan Priebe - Profihost AG]

Am 04.01.2018 um 08:27 schrieb Alexandre DERUMIER:
> does somebody have a redhat account to see te content of: 
> 
> https://access.redhat.com/solutions/3307851
> "Impacts of CVE-2017-5754, CVE-2017-5753, and CVE-2017-5715 to Red Hat 
> Virtualization products"

i don't have one but the content might be something like this:
https://www.suse.com/de-de/support/kb/doc/?id=7022512

So you need:
1.) intel / amd cpu microcode update
2.) qemu update to pass the new MSR and CPU flags from the microcode update
3.) host kernel update
4.) guest kernel update

The microcode update and the kernel update is publicly available but i'm
missing the qemu one.

Greets,
Stefan

> - Mail original -
> De: "aderumier" <aderum...@odiso.com>
> À: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag>
> Cc: "qemu-devel" <qemu-devel@nongnu.org>
> Envoyé: Jeudi 4 Janvier 2018 08:24:34
> Objet: Re: [Qemu-devel] CVE-2017-5715: relevant qemu patches
> 
>>> Can anybody point me to the relevant qemu patches? 
> 
> I don't have find them yet. 
> 
> Do you known if a vm using kvm64 cpu model is protected or not ? 
> 
> - Mail original - 
> De: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag> 
> À: "qemu-devel" <qemu-devel@nongnu.org> 
> Envoyé: Jeudi 4 Janvier 2018 07:27:01 
> Objet: [Qemu-devel] CVE-2017-5715: relevant qemu patches 
> 
> Hello, 
> 
> i've seen some vendors have updated qemu regarding meltdown / spectre. 
> 
> f.e.: 
> 
> CVE-2017-5715: QEMU was updated to allow passing through new MSR and 
> CPUID flags from the host VM to the CPU, to allow enabling/disabling 
> branch prediction features in the Intel CPU. (bsc#1068032) 
> 
> Can anybody point me to the relevant qemu patches? 
> 
> Thanks! 
> 
> Greets, 
> Stefan 
> 

[Qemu-devel] CVE-2017-5715: relevant qemu patches

[Stefan Priebe - Profihost AG]

Hello,

i've seen some vendors have updated qemu regarding meltdown / spectre.

f.e.:

 CVE-2017-5715: QEMU was updated to allow passing through new MSR and
 CPUID flags from the host VM to the CPU, to allow enabling/disabling
 branch prediction features in the Intel CPU. (bsc#1068032)

Can anybody point me to the relevant qemu patches?

Thanks!

Greets,
Stefan

Re: [Qemu-devel] dropped pkts with Qemu on tap interace (RX)

[Stefan Priebe - Profihost AG]

Am 03.01.2018 um 09:14 schrieb Alexandre DERUMIER:
> Hi Stefan,
> 
>>> The tap devices on the target vm shows dropped RX packages on BOTH tap 
>>> interfaces - strangely with the same amount of pkts? 
> 
> that's strange indeed. 
> if you tcpdump tap interfaces, do you see incoming traffic only on 1 
> interface, or both random ?

complete independend random traffic as it should.

> (can you provide the network configuration in the guest for both interfaces ?)

inside the guest? where the drop counter stays 0?

auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet static
address 192.168.0.2
netmask 255.255.255.0

that's it.

> I'm seeing that you have enable multiqueue on 1 of the interfaces, do you 
> have setup correctly the multiqueue part inside the guest.
uh oh? What is needed inside the guest?

> do you have enough vcpu to handle all the queues ?
Yes.

Stefan

> - Mail original -
> De: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag>
> À: "qemu-devel" <qemu-devel@nongnu.org>
> Envoyé: Mardi 2 Janvier 2018 12:17:29
> Objet: [Qemu-devel] dropped pkts with Qemu on tap interace (RX)
> 
> Hello, 
> 
> currently i'm trying to fix a problem where we have "random" missing 
> packets. 
> 
> We're doing an ssh connect from machine a to machine b every 5 minutes 
> via rsync and ssh. 
> 
> Sometimes it happens that we get this cron message: 
> "Connection to 192.168.0.2 closed by remote host. 
> rsync: connection unexpectedly closed (0 bytes received so far) [sender] 
> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] 
> ssh: connect to host 192.168.0.2 port 22: Connection refused" 
> 
> The tap devices on the target vm shows dropped RX packages on BOTH tap 
> interfaces - strangely with the same amount of pkts? 
> 
> # ifconfig tap317i0; ifconfig tap317i1 
> tap317i0 Link encap:Ethernet HWaddr 6e:cb:65:94:bb:bf 
> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 
> RX packets:2238445 errors:0 dropped:13159 overruns:0 frame:0 
> TX packets:9655853 errors:0 dropped:0 overruns:0 carrier:0 
> collisions:0 txqueuelen:1000 
> RX bytes:177991267 (169.7 MiB) TX bytes:910412749 (868.2 MiB) 
> 
> tap317i1 Link encap:Ethernet HWaddr 96:f8:b5:d0:9a:07 
> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 
> RX packets:1516085 errors:0 dropped:13159 overruns:0 frame:0 
> TX packets:1446964 errors:0 dropped:0 overruns:0 carrier:0 
> collisions:0 txqueuelen:1000 
> RX bytes:1597564313 (1.4 GiB) TX bytes:3517734365 (3.2 GiB) 
> 
> Any ideas how to inspect this issue? 
> 
> Greets, 
> Stefan 
> 

Re: [Qemu-devel] dropped pkts with Qemu on tap interace (RX)

[Stefan Priebe - Profihost AG]

Am 03.01.2018 um 04:57 schrieb Wei Xu:
> On Tue, Jan 02, 2018 at 10:17:25PM +0100, Stefan Priebe - Profihost AG wrote:
>>
>> Am 02.01.2018 um 18:04 schrieb Wei Xu:
>>> On Tue, Jan 02, 2018 at 04:24:33PM +0100, Stefan Priebe - Profihost AG 
>>> wrote:
>>>> Hi,
>>>> Am 02.01.2018 um 15:20 schrieb Wei Xu:
>>>>> On Tue, Jan 02, 2018 at 12:17:29PM +0100, Stefan Priebe - Profihost AG 
>>>>> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> currently i'm trying to fix a problem where we have "random" missing
>>>>>> packets.
>>>>>>
>>>>>> We're doing an ssh connect from machine a to machine b every 5 minutes
>>>>>> via rsync and ssh.
>>>>>>
>>>>>> Sometimes it happens that we get this cron message:
>>>>>> "Connection to 192.168.0.2 closed by remote host.
>>>>>> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
>>>>>> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
>>>>>> ssh: connect to host 192.168.0.2 port 22: Connection refused"
>>>>>
>>>>> Hi Stefan,
>>>>> What kind of virtio-net backend are you using? Can you paste your qemu
>>>>> command line here?
>>>>
>>>> Sure netdev part:
>>>> -netdev
>>>> type=tap,id=net0,ifname=tap317i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on
>>>> -device
>>>> virtio-net-pci,mac=EA:37:42:5C:F3:33,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
>>>> -netdev
>>>> type=tap,id=net1,ifname=tap317i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on,queues=4
>>>> -device
>>>> virtio-net-pci,mac=6A:8E:74:45:1A:0B,nedev=net1,bus=pci.0,addr=0x13,id=net1,vectors=10,mq=on,bootindex=301
>>>
>>> According to what you have mentioned, the traffic is not heavy for the 
>>> guests,
>>> the dropping shouldn't happen for regular case.
>>
>> The avg traffic is around 300kb/s.
>>
>>> What is your hardware platform?
>>
>> Dual Intel Xeon E5-2680 v4
>>
>>> and Which versions are you using for both
>>> guest/host kernel
>> Kernel v4.4.103
>>
>>> and qemu?
>> 2.9.1
>>
>>> Are there other VMs on the same host?
>> Yes.
> 
> What about the CPU load? 

Host:
80-90% Idle
LoadAvg: 6-7

VM:
97%-99% Idle

>>>>> 'Connection refused' usually means that the client gets a TCP Reset rather
>>>>> than losing packets, so this might not be a relevant issue.
>>>>
>>>> Mhm so you mean these might be two seperate ones?
>>>
>>> Yes.
>>>
>>>>
>>>>> Also you can do a tcpdump on both guests and see what happened to SSH 
>>>>> packets
>>>>> (tcpdump -i tapXXX port 22).
>>>>
>>>> Sadly not as there's too much traffic on that part as rsync is syncing
>>>> every 5 minutes through ssh.
>>>
>>> You can do a tcpdump for the entire traffic from the guest and host and 
>>> compare
>>> what kind of packets are dropped if the traffic is not overloaded.
>>
>> Are you sure? I don't get why the same amount and same kind of packets
>> should be received by both tap which are connected to different bridges
>> to different HW and physical interfaces.
> 
> Exactly, possibly this would be a host or guest kernel bug cos than qemu issue
> you are using vhost kernel as the backend and the two stats are independent,
> you might have to check out what is happening inside the traffic.

What do you mean by inside the traffic?

Stefan

Re: [Qemu-devel] dropped pkts with Qemu on tap interace (RX)

[Stefan Priebe - Profihost AG]

Am 02.01.2018 um 18:04 schrieb Wei Xu:
> On Tue, Jan 02, 2018 at 04:24:33PM +0100, Stefan Priebe - Profihost AG wrote:
>> Hi,
>> Am 02.01.2018 um 15:20 schrieb Wei Xu:
>>> On Tue, Jan 02, 2018 at 12:17:29PM +0100, Stefan Priebe - Profihost AG 
>>> wrote:
>>>> Hello,
>>>>
>>>> currently i'm trying to fix a problem where we have "random" missing
>>>> packets.
>>>>
>>>> We're doing an ssh connect from machine a to machine b every 5 minutes
>>>> via rsync and ssh.
>>>>
>>>> Sometimes it happens that we get this cron message:
>>>> "Connection to 192.168.0.2 closed by remote host.
>>>> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
>>>> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
>>>> ssh: connect to host 192.168.0.2 port 22: Connection refused"
>>>
>>> Hi Stefan,
>>> What kind of virtio-net backend are you using? Can you paste your qemu
>>> command line here?
>>
>> Sure netdev part:
>> -netdev
>> type=tap,id=net0,ifname=tap317i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on
>> -device
>> virtio-net-pci,mac=EA:37:42:5C:F3:33,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
>> -netdev
>> type=tap,id=net1,ifname=tap317i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on,queues=4
>> -device
>> virtio-net-pci,mac=6A:8E:74:45:1A:0B,nedev=net1,bus=pci.0,addr=0x13,id=net1,vectors=10,mq=on,bootindex=301
> 
> According to what you have mentioned, the traffic is not heavy for the guests,
> the dropping shouldn't happen for regular case.

The avg traffic is around 300kb/s.

> What is your hardware platform?

Dual Intel Xeon E5-2680 v4

> and Which versions are you using for both
> guest/host kernel
Kernel v4.4.103

> and qemu?
2.9.1

> Are there other VMs on the same host?
Yes.


>>> 'Connection refused' usually means that the client gets a TCP Reset rather
>>> than losing packets, so this might not be a relevant issue.
>>
>> Mhm so you mean these might be two seperate ones?
> 
> Yes.
> 
>>
>>> Also you can do a tcpdump on both guests and see what happened to SSH 
>>> packets
>>> (tcpdump -i tapXXX port 22).
>>
>> Sadly not as there's too much traffic on that part as rsync is syncing
>> every 5 minutes through ssh.
> 
> You can do a tcpdump for the entire traffic from the guest and host and 
> compare
> what kind of packets are dropped if the traffic is not overloaded.

Are you sure? I don't get why the same amount and same kind of packets
should be received by both tap which are connected to different bridges
to different HW and physical interfaces.

Stefan

> Wei
> 
>>
>>>> The tap devices on the target vm shows dropped RX packages on BOTH tap
>>>> interfaces - strangely with the same amount of pkts?
>>>>
>>>> # ifconfig tap317i0; ifconfig tap317i1
>>>> tap317i0  Link encap:Ethernet  HWaddr 6e:cb:65:94:bb:bf
>>>>   UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>>>>   RX packets:2238445 errors:0 dropped:13159 overruns:0 frame:0
>>>>   TX packets:9655853 errors:0 dropped:0 overruns:0 carrier:0
>>>>   collisions:0 txqueuelen:1000
>>>>   RX bytes:177991267 (169.7 MiB)  TX bytes:910412749 (868.2 MiB)
>>>>
>>>> tap317i1  Link encap:Ethernet  HWaddr 96:f8:b5:d0:9a:07
>>>>   UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>>>>   RX packets:1516085 errors:0 dropped:13159 overruns:0 frame:0
>>>>   TX packets:1446964 errors:0 dropped:0 overruns:0 carrier:0
>>>>   collisions:0 txqueuelen:1000
>>>>   RX bytes:1597564313 (1.4 GiB)  TX bytes:3517734365 (3.2 GiB)
>>>>
>>>> Any ideas how to inspect this issue?
>>>
>>> It seems both tap interfaces lose RX pkts, dropping pkts of RX means the
>>> host(backend) cann't receive packets from the guest as fast as the guest 
>>> sends.
>>
>> Inside the guest i see no dropped packets at all. It's only on the host
>> and strangely on both taps at the same value? And both are connected to
>> absolutely different networks.
>>
>>> Are you running some symmetrical test on both guests? 
>>
>> No.
>>
>> Stefan
>>
>>
>>> Wei
>>>
>>>>
>>>> Greets,
>>>> Stefan
>>>>

Re: [Qemu-devel] dropped pkts with Qemu on tap interace (RX)

[Stefan Priebe - Profihost AG]

Hi,
Am 02.01.2018 um 15:20 schrieb Wei Xu:
> On Tue, Jan 02, 2018 at 12:17:29PM +0100, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> currently i'm trying to fix a problem where we have "random" missing
>> packets.
>>
>> We're doing an ssh connect from machine a to machine b every 5 minutes
>> via rsync and ssh.
>>
>> Sometimes it happens that we get this cron message:
>> "Connection to 192.168.0.2 closed by remote host.
>> rsync: connection unexpectedly closed (0 bytes received so far) [sender]
>> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
>> ssh: connect to host 192.168.0.2 port 22: Connection refused"
> 
> Hi Stefan,
> What kind of virtio-net backend are you using? Can you paste your qemu
> command line here?

Sure netdev part:
-netdev
type=tap,id=net0,ifname=tap317i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on
-device
virtio-net-pci,mac=EA:37:42:5C:F3:33,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
-netdev
type=tap,id=net1,ifname=tap317i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on,queues=4
-device
virtio-net-pci,mac=6A:8E:74:45:1A:0B,nedev=net1,bus=pci.0,addr=0x13,id=net1,vectors=10,mq=on,bootindex=301


> 'Connection refused' usually means that the client gets a TCP Reset rather
> than losing packets, so this might not be a relevant issue.

Mhm so you mean these might be two seperate ones?

> Also you can do a tcpdump on both guests and see what happened to SSH packets
> (tcpdump -i tapXXX port 22).

Sadly not as there's too much traffic on that part as rsync is syncing
every 5 minutes through ssh.

>> The tap devices on the target vm shows dropped RX packages on BOTH tap
>> interfaces - strangely with the same amount of pkts?
>>
>> # ifconfig tap317i0; ifconfig tap317i1
>> tap317i0  Link encap:Ethernet  HWaddr 6e:cb:65:94:bb:bf
>>   UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>>   RX packets:2238445 errors:0 dropped:13159 overruns:0 frame:0
>>   TX packets:9655853 errors:0 dropped:0 overruns:0 carrier:0
>>   collisions:0 txqueuelen:1000
>>   RX bytes:177991267 (169.7 MiB)  TX bytes:910412749 (868.2 MiB)
>>
>> tap317i1  Link encap:Ethernet  HWaddr 96:f8:b5:d0:9a:07
>>   UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>>   RX packets:1516085 errors:0 dropped:13159 overruns:0 frame:0
>>   TX packets:1446964 errors:0 dropped:0 overruns:0 carrier:0
>>   collisions:0 txqueuelen:1000
>>   RX bytes:1597564313 (1.4 GiB)  TX bytes:3517734365 (3.2 GiB)
>>
>> Any ideas how to inspect this issue?
> 
> It seems both tap interfaces lose RX pkts, dropping pkts of RX means the
> host(backend) cann't receive packets from the guest as fast as the guest 
> sends.

Inside the guest i see no dropped packets at all. It's only on the host
and strangely on both taps at the same value? And both are connected to
absolutely different networks.

> Are you running some symmetrical test on both guests? 

No.

Stefan


> Wei
> 
>>
>> Greets,
>> Stefan
>>

[Qemu-devel] dropped pkts with Qemu on tap interace (RX)

[Stefan Priebe - Profihost AG]

Hello,

currently i'm trying to fix a problem where we have "random" missing
packets.

We're doing an ssh connect from machine a to machine b every 5 minutes
via rsync and ssh.

Sometimes it happens that we get this cron message:
"Connection to 192.168.0.2 closed by remote host.
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
ssh: connect to host 192.168.0.2 port 22: Connection refused"

The tap devices on the target vm shows dropped RX packages on BOTH tap
interfaces - strangely with the same amount of pkts?

# ifconfig tap317i0; ifconfig tap317i1
tap317i0  Link encap:Ethernet  HWaddr 6e:cb:65:94:bb:bf
  UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
  RX packets:2238445 errors:0 dropped:13159 overruns:0 frame:0
  TX packets:9655853 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:1000
  RX bytes:177991267 (169.7 MiB)  TX bytes:910412749 (868.2 MiB)

tap317i1  Link encap:Ethernet  HWaddr 96:f8:b5:d0:9a:07
  UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
  RX packets:1516085 errors:0 dropped:13159 overruns:0 frame:0
  TX packets:1446964 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:1000
  RX bytes:1597564313 (1.4 GiB)  TX bytes:3517734365 (3.2 GiB)

Any ideas how to inspect this issue?

Greets,
Stefan

Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support

[Stefan Priebe - Profihost AG]

Hello,

Am 22.11.2017 um 20:41 schrieb Dr. David Alan Gilbert:
> * Paolo Bonzini (pbonz...@redhat.com) wrote:
>> On 06/11/2017 12:09, Stefan Priebe - Profihost AG wrote:
>>> HI Paolo,
>>>
>>> could this patchset be related?
>>
>> Uh oh, yes it should.  Jason, any ways to fix it?  I suppose we need to
>> disable UFO in the newest machine types, but do we also have to do
>> (software) UFO in vhost-net and QEMU for migration compatibility?
> 
> Was there a solution to this?

it will be this one:
https://patchwork.ozlabs.org/patch/840094/


Stefan

> Dave
> 
>> Thanks,
>>
>> Paolo
>>
>>> Greets,
>>> Stefan
>>>
>>> Am 06.11.2017 um 10:52 schrieb Stefan Priebe - Profihost AG:
>>>> Hi Paolo,
>>>>
>>>> Am 06.11.2017 um 10:49 schrieb Paolo Bonzini:
>>>>> On 06/11/2017 10:48, Stefan Priebe - Profihost AG wrote:
>>>>>> Hi Paolo,
>>>>>>
>>>>>> Am 06.11.2017 um 10:40 schrieb Paolo Bonzini:
>>>>>>> On 06/11/2017 10:38, Stefan Priebe - Profihost AG wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu
>>>>>>>> 2.9.1.
>>>>>>>>
>>>>>>>> If i migrate a VM from a host running kernel 4.4 to a host running 4.12
>>>>>>>> i get:
>>>>>>>>
>>>>>>>> kvm: virtio-net: saved image requires TUN_F_UFO support
>>>>>>>> kvm: Failed to load virtio-net-device:tmp
>>>>>>>> kvm: Failed to load virtio-net:virtio
>>>>>>>> kvm: error while loading state for instance 0x0 of device
>>>>>>>> ':00:12.0/virtio-net'
>>>>>>>> kvm: load of migration failed: Invalid argument
>>>>>>>>
>>>>>>>>
>>>>>>>> while migrating from 4.12 to 4.4 works fine.
>>>>>>>>
>>>>>>>> Can anybody help? Is this expected?
>>>>>>>
>>>>>>> Can you check why peer_has_ufo failed (in hw/net/virtio-net.c)?
>>>>>>
>>>>>> May be - how can i archieve this? Patching the code is not a problem if
>>>>>> you can give me a hint.
>>>>>>
>>>>>>> Also, did this ioctl fail when the tap device was set up on the 4.12 
>>>>>>> destination?
>>>>>>> int tap_probe_has_ufo(int fd)
>>>>>>> {
>>>>>>> unsigned offload;
>>>>>>>
>>>>>>> offload = TUN_F_CSUM | TUN_F_UFO;
>>>>>>>
>>>>>>> if (ioctl(fd, TUNSETOFFLOAD, offload) < 0)
>>>>>>> return 0;
>>>>>>>
>>>>>>> return 1;
>>>>>>> }
>>>>>>
>>>>>> Should there be any kernel output or how can i detect / check it?
>>>>>
>>>>> For both, the simplest answer is probably just using printf.
>>>>
>>>> arg i missed an important part. The kernel is an opensuse SLE15 one.
>>>>
>>>> I've seen it contains the following patchset:
>>>> https://www.spinics.net/lists/netdev/msg443821.html
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>
>>
> --
> Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
> 

[Qemu-devel] kvm: Failed to flush the L2 table cache: Input/output error

[Stefan Priebe - Profihost AG]

Hello,

while using qemu 2.9.1 and doing a backup of a disk:

I have sometimes the following output:

Formatting '/mnt/qemu-249-2017_11_19-04_00_05.qcow2', fmt=qcow2
size=236223201280 encryption=off cluster_size=65536 lazy_refcounts=off
refcount_bits=16


followed by:

kvm: Failed to flush the L2 table cache: Input/output error
kvm: Failed to flush the refcount block cache: Input/output error

If this error happens (Failed to flush) the whole backup is incomplete.
Host kernel is a 4.4 based openSuSE 42.3 kernel.

Is this a known bug?

Greets,
Stefan

Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support

[Stefan Priebe - Profihost AG]

Hello,

Am 10.11.2017 um 05:18 schrieb Jason Wang:
> 
> 
> On 2017年11月08日 19:22, Jason Wang wrote:
>>
>>
>> On 2017年11月08日 18:46, Paolo Bonzini wrote:
>>> On 08/11/2017 09:21, Jason Wang wrote:
>>>>
>>>> On 2017年11月08日 17:05, Stefan Priebe - Profihost AG wrote:
>>>>> Am 08.11.2017 um 08:54 schrieb Jason Wang:
>>>>>> On 2017年11月08日 15:41, Stefan Priebe - Profihost AG wrote:
>>>>>>> Hi Paolo,
>>>>>>>
>>>>>>> Am 06.11.2017 um 12:27 schrieb Paolo Bonzini:
>>>>>>>> On 06/11/2017 12:09, Stefan Priebe - Profihost AG wrote:
>>>>>>>>> HI Paolo,
>>>>>>>>>
>>>>>>>>> could this patchset be related?
>>>>>>>> Uh oh, yes it should.  Jason, any ways to fix it?  I suppose we
>>>>>>>> need to
>>>>>>>> disable UFO in the newest machine types, but do we also have to do
>>>>>>>> (software) UFO in vhost-net and QEMU for migration compatibility?
>>>>>>> Any news on this?
>>>>>>>
>>>>>>> Greets,
>>>>>> Since we probe UFO support, it will be disabled automatically on
>>>>>> device
>>>>>> startup.
>>>>>>
>>>>>> For the issue of migration, I think the only way is trying to fix
>>>>>> it in
>>>>>> kernel.
>>>>> But 4.14 is around the corner and the patchset is already included.
>>>>> Shouldn't this get fixed before 4.14 is released?
>>>> We will try to seek a solution as soon as possible. If we can't catch
>>>> 4.14, we will do it for stable for sure.
>>> Jason, can you write to netdev and Cc Linus and me?
>>>
>>> Thanks,
>>>
>>> Paolo
>>
>> Paolo, see this https://www.spinics.net/lists/netdev/msg465454.html
>>
>> Just notice this today since I'm not on the cc list.
>>
>> Thanks
>>
>>
> 
> An update:
> 
> After some discussions on netdev, we will try to bring UFO back
> partially for just TAP. Willem promise to fix this. 4.14 is too late
> consider the fix is not trivial, it will go through stable tree.

OK is it save to just revert the UFO patchset for my local branch?

Greets,
Stefan

> 
> Thanks

Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support

[Stefan Priebe - Profihost AG]

Am 08.11.2017 um 08:54 schrieb Jason Wang:
> 
> 
> On 2017年11月08日 15:41, Stefan Priebe - Profihost AG wrote:
>> Hi Paolo,
>>
>> Am 06.11.2017 um 12:27 schrieb Paolo Bonzini:
>>> On 06/11/2017 12:09, Stefan Priebe - Profihost AG wrote:
>>>> HI Paolo,
>>>>
>>>> could this patchset be related?
>>> Uh oh, yes it should.  Jason, any ways to fix it?  I suppose we need to
>>> disable UFO in the newest machine types, but do we also have to do
>>> (software) UFO in vhost-net and QEMU for migration compatibility?
>> Any news on this?
>>
>> Greets,
> 
> Since we probe UFO support, it will be disabled automatically on device
> startup.
> 
> For the issue of migration, I think the only way is trying to fix it in
> kernel.

But 4.14 is around the corner and the patchset is already included.
Shouldn't this get fixed before 4.14 is released?

Stefan
> 
> Thanks

Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support

[Stefan Priebe - Profihost AG]

Hi Paolo,

Am 06.11.2017 um 12:27 schrieb Paolo Bonzini:
> On 06/11/2017 12:09, Stefan Priebe - Profihost AG wrote:
>> HI Paolo,
>>
>> could this patchset be related?
> 
> Uh oh, yes it should.  Jason, any ways to fix it?  I suppose we need to
> disable UFO in the newest machine types, but do we also have to do
> (software) UFO in vhost-net and QEMU for migration compatibility?

Any news on this?

Greets,
Stefan


> Thanks,
> 
> Paolo
> 
>> Greets,
>> Stefan
>>
>> Am 06.11.2017 um 10:52 schrieb Stefan Priebe - Profihost AG:
>>> Hi Paolo,
>>>
>>> Am 06.11.2017 um 10:49 schrieb Paolo Bonzini:
>>>> On 06/11/2017 10:48, Stefan Priebe - Profihost AG wrote:
>>>>> Hi Paolo,
>>>>>
>>>>> Am 06.11.2017 um 10:40 schrieb Paolo Bonzini:
>>>>>> On 06/11/2017 10:38, Stefan Priebe - Profihost AG wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu
>>>>>>> 2.9.1.
>>>>>>>
>>>>>>> If i migrate a VM from a host running kernel 4.4 to a host running 4.12
>>>>>>> i get:
>>>>>>>
>>>>>>> kvm: virtio-net: saved image requires TUN_F_UFO support
>>>>>>> kvm: Failed to load virtio-net-device:tmp
>>>>>>> kvm: Failed to load virtio-net:virtio
>>>>>>> kvm: error while loading state for instance 0x0 of device
>>>>>>> ':00:12.0/virtio-net'
>>>>>>> kvm: load of migration failed: Invalid argument
>>>>>>>
>>>>>>>
>>>>>>> while migrating from 4.12 to 4.4 works fine.
>>>>>>>
>>>>>>> Can anybody help? Is this expected?
>>>>>>
>>>>>> Can you check why peer_has_ufo failed (in hw/net/virtio-net.c)?
>>>>>
>>>>> May be - how can i archieve this? Patching the code is not a problem if
>>>>> you can give me a hint.
>>>>>
>>>>>> Also, did this ioctl fail when the tap device was set up on the 4.12 
>>>>>> destination?
>>>>>> int tap_probe_has_ufo(int fd)
>>>>>> {
>>>>>> unsigned offload;
>>>>>>
>>>>>> offload = TUN_F_CSUM | TUN_F_UFO;
>>>>>>
>>>>>> if (ioctl(fd, TUNSETOFFLOAD, offload) < 0)
>>>>>> return 0;
>>>>>>
>>>>>> return 1;
>>>>>> }
>>>>>
>>>>> Should there be any kernel output or how can i detect / check it?
>>>>
>>>> For both, the simplest answer is probably just using printf.
>>>
>>> arg i missed an important part. The kernel is an opensuse SLE15 one.
>>>
>>> I've seen it contains the following patchset:
>>> https://www.spinics.net/lists/netdev/msg443821.html
>>>
>>> Greets,
>>> Stefan
>>>
> 

Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support

[Stefan Priebe - Profihost AG]

HI Paolo,

could this patchset be related?

Greets,
Stefan

Am 06.11.2017 um 10:52 schrieb Stefan Priebe - Profihost AG:
> Hi Paolo,
> 
> Am 06.11.2017 um 10:49 schrieb Paolo Bonzini:
>> On 06/11/2017 10:48, Stefan Priebe - Profihost AG wrote:
>>> Hi Paolo,
>>>
>>> Am 06.11.2017 um 10:40 schrieb Paolo Bonzini:
>>>> On 06/11/2017 10:38, Stefan Priebe - Profihost AG wrote:
>>>>> Hello,
>>>>>
>>>>> i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu
>>>>> 2.9.1.
>>>>>
>>>>> If i migrate a VM from a host running kernel 4.4 to a host running 4.12
>>>>> i get:
>>>>>
>>>>> kvm: virtio-net: saved image requires TUN_F_UFO support
>>>>> kvm: Failed to load virtio-net-device:tmp
>>>>> kvm: Failed to load virtio-net:virtio
>>>>> kvm: error while loading state for instance 0x0 of device
>>>>> ':00:12.0/virtio-net'
>>>>> kvm: load of migration failed: Invalid argument
>>>>>
>>>>>
>>>>> while migrating from 4.12 to 4.4 works fine.
>>>>>
>>>>> Can anybody help? Is this expected?
>>>>
>>>> Can you check why peer_has_ufo failed (in hw/net/virtio-net.c)?
>>>
>>> May be - how can i archieve this? Patching the code is not a problem if
>>> you can give me a hint.
>>>
>>>> Also, did this ioctl fail when the tap device was set up on the 4.12 
>>>> destination?
>>>> int tap_probe_has_ufo(int fd)
>>>> {
>>>> unsigned offload;
>>>>
>>>> offload = TUN_F_CSUM | TUN_F_UFO;
>>>>
>>>> if (ioctl(fd, TUNSETOFFLOAD, offload) < 0)
>>>> return 0;
>>>>
>>>> return 1;
>>>> }
>>>
>>> Should there be any kernel output or how can i detect / check it?
>>
>> For both, the simplest answer is probably just using printf.
> 
> arg i missed an important part. The kernel is an opensuse SLE15 one.
> 
> I've seen it contains the following patchset:
> https://www.spinics.net/lists/netdev/msg443821.html
> 
> Greets,
> Stefan
> 

Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support

[Stefan Priebe - Profihost AG]

Hi Paolo,

Am 06.11.2017 um 10:49 schrieb Paolo Bonzini:
> On 06/11/2017 10:48, Stefan Priebe - Profihost AG wrote:
>> Hi Paolo,
>>
>> Am 06.11.2017 um 10:40 schrieb Paolo Bonzini:
>>> On 06/11/2017 10:38, Stefan Priebe - Profihost AG wrote:
>>>> Hello,
>>>>
>>>> i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu
>>>> 2.9.1.
>>>>
>>>> If i migrate a VM from a host running kernel 4.4 to a host running 4.12
>>>> i get:
>>>>
>>>> kvm: virtio-net: saved image requires TUN_F_UFO support
>>>> kvm: Failed to load virtio-net-device:tmp
>>>> kvm: Failed to load virtio-net:virtio
>>>> kvm: error while loading state for instance 0x0 of device
>>>> ':00:12.0/virtio-net'
>>>> kvm: load of migration failed: Invalid argument
>>>>
>>>>
>>>> while migrating from 4.12 to 4.4 works fine.
>>>>
>>>> Can anybody help? Is this expected?
>>>
>>> Can you check why peer_has_ufo failed (in hw/net/virtio-net.c)?
>>
>> May be - how can i archieve this? Patching the code is not a problem if
>> you can give me a hint.
>>
>>> Also, did this ioctl fail when the tap device was set up on the 4.12 
>>> destination?
>>> int tap_probe_has_ufo(int fd)
>>> {
>>> unsigned offload;
>>>
>>> offload = TUN_F_CSUM | TUN_F_UFO;
>>>
>>> if (ioctl(fd, TUNSETOFFLOAD, offload) < 0)
>>> return 0;
>>>
>>> return 1;
>>> }
>>
>> Should there be any kernel output or how can i detect / check it?
> 
> For both, the simplest answer is probably just using printf.

arg i missed an important part. The kernel is an opensuse SLE15 one.

I've seen it contains the following patchset:
https://www.spinics.net/lists/netdev/msg443821.html

Greets,
Stefan

Re: [Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support

[Stefan Priebe - Profihost AG]

Hi Paolo,

Am 06.11.2017 um 10:40 schrieb Paolo Bonzini:
> On 06/11/2017 10:38, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu
>> 2.9.1.
>>
>> If i migrate a VM from a host running kernel 4.4 to a host running 4.12
>> i get:
>>
>> kvm: virtio-net: saved image requires TUN_F_UFO support
>> kvm: Failed to load virtio-net-device:tmp
>> kvm: Failed to load virtio-net:virtio
>> kvm: error while loading state for instance 0x0 of device
>> ':00:12.0/virtio-net'
>> kvm: load of migration failed: Invalid argument
>>
>>
>> while migrating from 4.12 to 4.4 works fine.
>>
>> Can anybody help? Is this expected?
> 
> Can you check why peer_has_ufo failed (in hw/net/virtio-net.c)?

May be - how can i archieve this? Patching the code is not a problem if
you can give me a hint.

> Also, did this ioctl fail when the tap device was set up on the 4.12 
> destination?
> int tap_probe_has_ufo(int fd)
> {
> unsigned offload;
> 
> offload = TUN_F_CSUM | TUN_F_UFO;
> 
> if (ioctl(fd, TUNSETOFFLOAD, offload) < 0)
> return 0;
> 
> return 1;
> }

Should there be any kernel output or how can i detect / check it?

Greets,
Stefan

> Thanks,
> 
> Paolo
> 

[Qemu-devel] kvm: virtio-net: saved image requires TUN_F_UFO support

[Stefan Priebe - Profihost AG]

Hello,

i've upgraded some servers from kernel 4.4 to 4.12 - both running Qemu
2.9.1.

If i migrate a VM from a host running kernel 4.4 to a host running 4.12
i get:

kvm: virtio-net: saved image requires TUN_F_UFO support
kvm: Failed to load virtio-net-device:tmp
kvm: Failed to load virtio-net:virtio
kvm: error while loading state for instance 0x0 of device
':00:12.0/virtio-net'
kvm: load of migration failed: Invalid argument


while migrating from 4.12 to 4.4 works fine.

Can anybody help? Is this expected?

Greets,
Stefan

Re: [Qemu-devel] virtio: network: lost tcp/ip packets

[Stefan Priebe - Profihost AG]

Hello Stefan,


Am 30.08.2017 um 19:17 schrieb Stefan Hajnoczi:
> On Fri, Aug 18, 2017 at 04:40:36PM +0200, Stefan Priebe - Profihost AG wrote:
>> i've a problem with two VMs running on the SAME host machine using qemu
>> 2.7.1 or 2.9.0 and vhost_net + virtio.
>>
>> Sometimes TCP packets going from machine a to machine b are simply lost.
>> I see them in VM A using tcpdump going out but they never come in on
>> machine B. Both machines have iptables and stuff turned off.
>>
>> On the host both VMs network interfaces are connected to the same bridge.
>>
>> Any ideas? May be a known bug?
>>
>> Host and Guest Kernel is an OpenSuSE 42.3 kernel based on vanilla 4.4.82.
> 
> Have you tcpdumped the tap interfaces on the host?
> 
currently i can't reproduce the issue - even i was for two days.
Currently i've no idea what has happend and will report back if it
happens again.

Thanks!

Greets,
Stefan

Re: [Qemu-devel] virtio: network: lost tcp/ip packets

[Stefan Priebe - Profihost AG]

Hello,

does nobody have an idea?

Greets,
Stefam

Am 18.08.2017 um 16:40 schrieb Stefan Priebe - Profihost AG:
> Hello,
> 
> i've a problem with two VMs running on the SAME host machine using qemu
> 2.7.1 or 2.9.0 and vhost_net + virtio.
> 
> Sometimes TCP packets going from machine a to machine b are simply lost.
> I see them in VM A using tcpdump going out but they never come in on
> machine B. Both machines have iptables and stuff turned off.
> 
> On the host both VMs network interfaces are connected to the same bridge.
> 
> Any ideas? May be a known bug?
> 
> Host and Guest Kernel is an OpenSuSE 42.3 kernel based on vanilla 4.4.82.
> 
> Thanks!
> 
> Greets,
> Stefan
> 

[Qemu-devel] virtio: network: lost tcp/ip packets

[Stefan Priebe - Profihost AG]

Hello,

i've a problem with two VMs running on the SAME host machine using qemu
2.7.1 or 2.9.0 and vhost_net + virtio.

Sometimes TCP packets going from machine a to machine b are simply lost.
I see them in VM A using tcpdump going out but they never come in on
machine B. Both machines have iptables and stuff turned off.

On the host both VMs network interfaces are connected to the same bridge.

Any ideas? May be a known bug?

Host and Guest Kernel is an OpenSuSE 42.3 kernel based on vanilla 4.4.82.

Thanks!

Greets,
Stefan

Re: [Qemu-devel] any known virtio-net regressions in Qemu 2.7?

[Stefan Priebe - Profihost AG]

Am 19.12.2016 um 12:03 schrieb Stefan Hajnoczi:
> On Fri, Dec 16, 2016 at 10:00:36PM +0100, Stefan Priebe - Profihost AG wrote:
>>
>> Am 15.12.2016 um 07:46 schrieb Alexandre DERUMIER:
>>> does rollbacking the kernel to previous version fix the problem ?
>>
>> The culprit is the used tuned agent from Redhat
>> (https://github.com/redhat-performance/tuned). The used profile
>> virtual-host results in these problems. Stopping tuned or using another
>> profile like throughput-performance everything is fine again.
> 
> Interesting discovery.  Have you filed a bug report about it?

As we're not using RHEL, nor CentOS i didn't expect this to make sense
;-) I just believed that something in the kernel changed.

Greets,
Stefan

>> after upgrading a cluster OS, Qemu, ... i'm experiencing slow and
>> volatile network speeds inside my VMs.
>>
>> Currently I've no idea what causes this but it's related to the host
>> upgrades. Before i was running Qemu 2.6.2.
>>
>> I'm using virtio for the network cards.
> 
> Stefan
> 

Re: [Qemu-devel] any known virtio-net regressions in Qemu 2.7?

[Stefan Priebe - Profihost AG]

Am 15.12.2016 um 07:46 schrieb Alexandre DERUMIER:
> does rollbacking the kernel to previous version fix the problem ?

The culprit is the used tuned agent from Redhat
(https://github.com/redhat-performance/tuned). The used profile
virtual-host results in these problems. Stopping tuned or using another
profile like throughput-performance everything is fine again.

Geets,
Stefan

> 
> i'm not sure if "perf" could give you some hints
> - Mail original -
> De: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag>
> À: "aderumier" <aderum...@odiso.com>
> Cc: "qemu-devel" <qemu-devel@nongnu.org>
> Envoyé: Mercredi 14 Décembre 2016 21:36:23
> Objet: Re: [Qemu-devel] any known virtio-net regressions in Qemu 2.7?
> 
> Am 14.12.2016 um 16:33 schrieb Alexandre DERUMIER: 
>> Hi Stefan, 
>>
>> do you have upgraded kernel ? 
> 
> Yes sure. But i'm out of ideas how to debug. Sometimes it gives me 
> constant 80MB/s, sometimes 125 and sometimes only 6. While on the host 
> the cards are not busy. 
> 
> Greets, 
> Stefan 
> 
>>
>> maybe it could be related to vhost-net module too. 
>>
>>
>> - Mail original - 
>> De: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag> 
>> À: "qemu-devel" <qemu-devel@nongnu.org> 
>> Envoyé: Mercredi 14 Décembre 2016 16:04:08 
>> Objet: [Qemu-devel] any known virtio-net regressions in Qemu 2.7? 
>>
>> Hello, 
>>
>> after upgrading a cluster OS, Qemu, ... i'm experiencing slow and 
>> volatile network speeds inside my VMs. 
>>
>> Currently I've no idea what causes this but it's related to the host 
>> upgrades. Before i was running Qemu 2.6.2. 
>>
>> I'm using virtio for the network cards. 
>>
>> Greets, 
>> Stefan 
>>
> 

Re: [Qemu-devel] any known virtio-net regressions in Qemu 2.7?

[Stefan Priebe - Profihost AG]

Am 14.12.2016 um 16:33 schrieb Alexandre DERUMIER:
> Hi Stefan,
> 
> do you have upgraded kernel ?

Yes sure. But i'm out of ideas how to debug. Sometimes it gives me
constant 80MB/s, sometimes 125 and sometimes only 6. While on the host
the cards are not busy.

Greets,
Stefan

> 
> maybe it could be related to vhost-net module too.
> 
> 
> - Mail original -----
> De: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag>
> À: "qemu-devel" <qemu-devel@nongnu.org>
> Envoyé: Mercredi 14 Décembre 2016 16:04:08
> Objet: [Qemu-devel] any known virtio-net regressions in Qemu 2.7?
> 
> Hello, 
> 
> after upgrading a cluster OS, Qemu, ... i'm experiencing slow and 
> volatile network speeds inside my VMs. 
> 
> Currently I've no idea what causes this but it's related to the host 
> upgrades. Before i was running Qemu 2.6.2. 
> 
> I'm using virtio for the network cards. 
> 
> Greets, 
> Stefan 
> 

[Qemu-devel] any known virtio-net regressions in Qemu 2.7?

[Stefan Priebe - Profihost AG]

Hello,

after upgrading a cluster OS, Qemu, ... i'm experiencing slow and
volatile network speeds inside my VMs.

Currently I've no idea what causes this but it's related to the host
upgrades. Before i was running Qemu 2.6.2.

I'm using virtio for the network cards.

Greets,
Stefan

Re: [Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Stefan Priebe - Profihost AG]

Am 15.11.2016 um 12:07 schrieb Ladi Prosek:
> Hi,
> 
> On Tue, Nov 15, 2016 at 11:37 AM, Stefan Priebe - Profihost AG
> <s.pri...@profihost.ag> wrote:
>> Hello,
>>
>> Am 15.11.2016 um 11:30 schrieb Dr. David Alan Gilbert:
>>> * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote:
>>>> Hello,
>>>>
>>>> today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM
>>>> is running windows and virtio-balloon and with machine type pc-i440fx-2.5.
>>>>
>>>> The output of the target qemu process was:
>>>> kvm_apic_post_load: Yeh
>>>> kvm_apic_post_load: Yeh
>>>> kvm_apic_post_load: Yeh
>>>> kvm_apic_post_load: Yeh
>>>> kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4
>>>> kvm: error while loading state for instance 0x0 of device
>>>> ':00:03.0/virtio-balloon'
>>>> kvm: load of migration failed: Operation not permitted
>>>
>>> Yes that's a known bug; only seems to affect windows guests, and I believe
>>> doesn't even need to cross versions.
>>>
>>> There's a bunch of fixes that Stefan applied to virtio code
>>> that I think fix this; I see that he cc'd qemu-stable.
>>> I think it's 4b7f91ed, but I'm not sure if there are others needed.
>>
>> thanks for pointing to that commit.
>>
>> Stefan can you tell me whether it's enough to cherry-pick 4b7f91ed into
>> 2.7.0 ?
> 
> I don't believe that 4b7f91ed will help here (no device reset on
> migration). We've seen this error with QEMU running without:
> 
>   commit 4eae2a657d1ff5ada56eb9b4966eae0eff333b0b
>   Author: Ladi Prosek <lpro...@redhat.com>
>   Date:   Tue Mar 1 12:14:03 2016 +0100
> 
>   balloon: fix segfault and harden the stats queue
> 
> 
> Is it possible that the VM has run on such a QEMU, then was
> live-migrated to 2.6.2, and then to 2.7.0?

Hi,

yes, it was started under Qemu 2.5.0. Was then migrated to 2.6.2 and
then to 2.7.0.

Greets,
Stefan

> 
> Thanks,
> Ladi
> 
>> Greets,
>> Stefan
>>
>>>
>>> Dave
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
>>>
>>

Re: [Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Stefan Priebe - Profihost AG]

Hello,

Am 15.11.2016 um 11:30 schrieb Dr. David Alan Gilbert:
> * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote:
>> Hello,
>>
>> today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM
>> is running windows and virtio-balloon and with machine type pc-i440fx-2.5.
>>
>> The output of the target qemu process was:
>> kvm_apic_post_load: Yeh
>> kvm_apic_post_load: Yeh
>> kvm_apic_post_load: Yeh
>> kvm_apic_post_load: Yeh
>> kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4
>> kvm: error while loading state for instance 0x0 of device
>> ':00:03.0/virtio-balloon'
>> kvm: load of migration failed: Operation not permitted
> 
> Yes that's a known bug; only seems to affect windows guests, and I believe
> doesn't even need to cross versions.
> 
> There's a bunch of fixes that Stefan applied to virtio code
> that I think fix this; I see that he cc'd qemu-stable.
> I think it's 4b7f91ed, but I'm not sure if there are others needed.

thanks for pointing to that commit.

Stefan can you tell me whether it's enough to cherry-pick 4b7f91ed into
2.7.0 ?

Greets,
Stefan

> 
> Dave
>>
>> Greets,
>> Stefan
>>
> --
> Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
> 

[Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Stefan Priebe - Profihost AG]

Hello,

today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM
is running windows and virtio-balloon and with machine type pc-i440fx-2.5.

The output of the target qemu process was:
kvm_apic_post_load: Yeh
kvm_apic_post_load: Yeh
kvm_apic_post_load: Yeh
kvm_apic_post_load: Yeh
kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4
kvm: error while loading state for instance 0x0 of device
':00:03.0/virtio-balloon'
kvm: load of migration failed: Operation not permitted

Greets,
Stefan

Re: [Qemu-devel] [PATCH] net: fix qemu_announce_self not emitting packets

[Stefan Priebe - Profihost AG]

Am 07.06.2016 um 09:37 schrieb Peter Lieven:
> commit fefe2a78 accidently dropped the code path for injecting
> raw packets. This feature is needed for sending gratuitous ARPs
> after an incoming migration has completed. The result is increased
> network downtime for vservers where the network card is not virtio-net
> with the VIRTIO_NET_F_GUEST_ANNOUNCE feature.
> 
> Fixes: fefe2a78abde932e0f340b21bded2c86def1d242
> Cc: qemu-sta...@nongnu.org
> Cc: yan...@cn.fujitsu.com
> Signed-off-by: Peter Lieven 
> ---
>  net/net.c | 6 +-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/net/net.c b/net/net.c
> index 5f3e5a9..d5834ea 100644
> --- a/net/net.c
> +++ b/net/net.c
> @@ -722,7 +722,11 @@ ssize_t qemu_deliver_packet_iov(NetClientState *sender,
>  return 0;
>  }
>  
> -if (nc->info->receive_iov) {
> +if (flags & QEMU_NET_PACKET_FLAG_RAW && iovcnt == 1 &&
> +nc->info->receive_raw) {
> +/* this is required for qemu_announce_self() */
> +ret = nc->info->receive_raw(nc, iov[0].iov_base, iov[0].iov_len);
> +} else  if (nc->info->receive_iov) {
>  ret = nc->info->receive_iov(nc, iov, iovcnt);
>  } else {
>  ret = nc_sendv_compat(nc, iov, iovcnt, flags);
> 

Thanks for the patch. Sadly it does not fix our problem. So we still
have another Problem.

Thanks!

Greets,
Stefan

Re: [Qemu-devel] [Qemu-stable] Broken live Migration in Qemu 2.5.1.1?

[Stefan Priebe - Profihost AG]

Am 07.06.2016 um 09:38 schrieb Peter Lieven:
> Am 06.06.2016 um 18:13 schrieb Stefan Priebe - Profihost AG:
>> We're most probably seeing the same while migrating a machine running
>> balanceng but haven't thought this might be a qemu bug. Instead we're
>> investigating with balanceng people.
>>
>> Waiting for your further results.
> 
> Can you try the patch
> 
> net: fix qemu_announce_self not emitting packets

Thx - will wait for a new patch based on Paolo's comments - if that's ok.

Stefan

> I just send to the mailing list.
> 
> Thanks,
> Peter
> 

Re: [Qemu-devel] [Qemu-stable] Broken live Migration in Qemu 2.5.1.1?

[Stefan Priebe - Profihost AG]

We're most probably seeing the same while migrating a machine running balanceng 
but haven't thought this might be a qemu bug. Instead we're investigating with 
balanceng people.

Waiting for your further results.

Greets,
Stefan

Excuse my typo sent from my mobile phone.

> Am 06.06.2016 um 17:51 schrieb Peter Lieven :
> 
>> Am 06.06.2016 um 15:32 schrieb Peter Lieven:
>> Hi,
>> 
>> during internal testing of Qemu 2.5.1.1 I found a vServer running Ubuntu 
>> 12.04 (kernel 3.13) and a slave SQL server to
>> stop replicating from the master. This seems to be reproducible. It is 
>> possible to continue replication when issuing a slave stop / slave start.
>> There is no error visible on the vServer.
>> 
>> Has anyone a fix in mind that could be related to such an issue?
>> 
>> Host kernel in Linux 4.4, Guest kernel 3.13. Guest driver is virtio-blk via 
>> iSCSI. Emulated vCPU is Westmere.
> 
> After a lot of testing I found out that obviously thats no block driver 
> problem, but a regression in the virtio-net or the network stack.
> 
> qemu_announce_self() is generating packets for all NICs, but it seems they 
> are no longer emitted. This worked at least in qemu-2.2.0 with
> the same guest kernel and host kernel.
> 
> I will continue debugging tomorrow why this happens.
> 
> Peter
> 

Re: [Qemu-devel] drive-backup

[Stefan Priebe]

Am 25.02.2016 um 20:53 schrieb John Snow:



On 02/25/2016 02:49 AM, Stefan Priebe - Profihost AG wrote:


Am 22.02.2016 um 23:08 schrieb John Snow:



On 02/22/2016 03:21 PM, Stefan Priebe wrote:

Hello,

is there any chance or hack to work with a bigger cluster size for the
drive backup job?

See:
http://git.qemu.org/?p=qemu.git;a=blob;f=block/backup.c;h=16105d40b193be9bb40346027bdf58e62b956a96;hb=98d2c6f2cd80afaa2dc10091f5e35a97c181e4f5


This is very slow with ceph - may be due to the 64k block size. I would
like to check whether this is faster with cephs native block size of 4mb.

Greets,
Stefan



It's hardcoded to 64K at the moment, but I am checking in a patch to
round up the cluster size to be the bigger of (64k,
$target_cluster_size) in order to make sure that incremental backups in
particular never copy a fraction of a cluster. As a side-effect, the
same round-up will happen for all modes (sync=top,none,full).

If QEMU is aware of the target cluster size of 4MB, this would
immediately jump the copy-size up to 4MB clusters for you.

See: https://lists.nongnu.org/archive/html/qemu-devel/2016-02/msg02839.html


Thanks for your patches and thanks for your great answer. But our
problem is not the target but the source ;-) The target has a local
cache and don't care about the cluster size but the source does not.

But it works fine if we change the default cluster size to 4MB. So it
has point us to the right direction.

Stefan



Ah, sorry, I misunderstood.

It's easy to change anyway! I am in favor of adding a configurable
parameter, as long as it keeps the other constraints I mentioned in mind.


ah great and thanks!

Stefan



--js



Otherwise, after my trivial fix, you should find cluster_size to be a
mutable concept and perhaps one that you could introduce a runtime
parameter for if you could convince the necessary parties that it's
needed in the API.

You'd have to be careful in the case of incremental that all the various
cluster sizes work well together:

- Granularity of bitmap (Defaults to cluster size of source, or 64K if
unknown. can be configured to be arbitrarily larger.)
- Cluster size of source file (For qcow2, defaults to 64k)
- Cluster size of target file
- Cluster size of backup routine (Currently always 64K)

I think that LCM(source_cluster_size, target_cluster_size,
backup_cluster_size) = MAX(A, B, C) will always be a safe minimum.

Bitmap granularity is more flexible, and it is more efficient when the
bitmap granularity matches the backup cluster size, but it can cope with
mismatches. If the bitmap is smaller or larger than the backup cluster
size, it generally means that more data that is clean is going to be
transferred across the pipe.

(Hmm, I wonder if it's worth checking in code to adjust a bitmap
granularity after it has been created so people can easily experiment
with performance tweaking here.)


Let me know if any of my ramble sounds interesting :)
--John

Re: [Qemu-devel] drive-backup

[Stefan Priebe - Profihost AG]

Am 22.02.2016 um 23:08 schrieb John Snow:
> 
> 
> On 02/22/2016 03:21 PM, Stefan Priebe wrote:
>> Hello,
>>
>> is there any chance or hack to work with a bigger cluster size for the
>> drive backup job?
>>
>> See:
>> http://git.qemu.org/?p=qemu.git;a=blob;f=block/backup.c;h=16105d40b193be9bb40346027bdf58e62b956a96;hb=98d2c6f2cd80afaa2dc10091f5e35a97c181e4f5
>>
>>
>> This is very slow with ceph - may be due to the 64k block size. I would
>> like to check whether this is faster with cephs native block size of 4mb.
>>
>> Greets,
>> Stefan
>>
> 
> It's hardcoded to 64K at the moment, but I am checking in a patch to
> round up the cluster size to be the bigger of (64k,
> $target_cluster_size) in order to make sure that incremental backups in
> particular never copy a fraction of a cluster. As a side-effect, the
> same round-up will happen for all modes (sync=top,none,full).
> 
> If QEMU is aware of the target cluster size of 4MB, this would
> immediately jump the copy-size up to 4MB clusters for you.
> 
> See: https://lists.nongnu.org/archive/html/qemu-devel/2016-02/msg02839.html

Thanks for your patches and thanks for your great answer. But our
problem is not the target but the source ;-) The target has a local
cache and don't care about the cluster size but the source does not.

But it works fine if we change the default cluster size to 4MB. So it
has point us to the right direction.

Stefan

> 
> Otherwise, after my trivial fix, you should find cluster_size to be a
> mutable concept and perhaps one that you could introduce a runtime
> parameter for if you could convince the necessary parties that it's
> needed in the API.
> 
> You'd have to be careful in the case of incremental that all the various
> cluster sizes work well together:
> 
> - Granularity of bitmap (Defaults to cluster size of source, or 64K if
> unknown. can be configured to be arbitrarily larger.)
> - Cluster size of source file (For qcow2, defaults to 64k)
> - Cluster size of target file
> - Cluster size of backup routine (Currently always 64K)
> 
> I think that LCM(source_cluster_size, target_cluster_size,
> backup_cluster_size) = MAX(A, B, C) will always be a safe minimum.
> 
> Bitmap granularity is more flexible, and it is more efficient when the
> bitmap granularity matches the backup cluster size, but it can cope with
> mismatches. If the bitmap is smaller or larger than the backup cluster
> size, it generally means that more data that is clean is going to be
> transferred across the pipe.
> 
> (Hmm, I wonder if it's worth checking in code to adjust a bitmap
> granularity after it has been created so people can easily experiment
> with performance tweaking here.)
> 
> 
> Let me know if any of my ramble sounds interesting :)
> --John
> 

[Qemu-devel] drive-backup

[Stefan Priebe]

Hello,

is there any chance or hack to work with a bigger cluster size for the 
drive backup job?


See:
http://git.qemu.org/?p=qemu.git;a=blob;f=block/backup.c;h=16105d40b193be9bb40346027bdf58e62b956a96;hb=98d2c6f2cd80afaa2dc10091f5e35a97c181e4f5

This is very slow with ceph - may be due to the 64k block size. I would 
like to check whether this is faster with cephs native block size of 4mb.


Greets,
Stefan

Re: [Qemu-devel] kernel 4.4.2: kvm_irq_delivery_to_api / rwsem_down_read_failed

[Stefan Priebe]

Am 22.02.2016 um 18:36 schrieb Paolo Bonzini:



On 20/02/2016 11:44, Stefan Priebe wrote:

Hi,

while testing Kernel 4.4.2 and starting 20 Qemu 2.4.1 virtual machines.
I got those traces and a load of 500 on those system. I was only abler
to recover by sysrq-trigger.


It seems like something happening at the VM level.  A task took the mm
semaphore and hung everyone else.  Difficult to debug without a core
(and without knowing who held the semaphore).  Sorry.


OK thank you anyway. Is there anything i can do if this happens again?

Stefan


Paolo



All traces:

INFO: task pvedaemon worke:7470 blocked for more than 120 seconds.
Not tainted 4.4.2+1-ph #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pvedaemon worke D 88239c367ca0 0 7470 7468 0x0008
88239c367ca0 8840a6232500 8823ed83a500 88239c367c90
88239c368000 8845f5f070e8 8845f5f07100 
7ffc73b48e58 88239c367cc0 b66a4d89 88239c367cf0
Call Trace:
[] schedule+0x39/0x80
[] rwsem_down_read_failed+0xc7/0x120
[] call_rwsem_down_read_failed+0x14/0x30
[] ? down_read+0x17/0x20
[] __access_remote_vm+0x3e/0x1c0
[] ? call_rwsem_down_read_failed+0x14/0x30
[] access_remote_vm+0x1f/0x30
[] proc_pid_cmdline_read+0x16e/0x4f0
[] ? acct_account_cputime+0x1c/0x20
[] __vfs_read+0x18/0x40
[] vfs_read+0x8e/0x140
[] SyS_read+0x4f/0xa0
[] entry_SYSCALL_64_fastpath+0x12/0x71
INFO: task pvestatd:7633 blocked for more than 120 seconds.
Not tainted 4.4.2+1-ph #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pvestatd D 88239f16fd40 0 7633 1 0x0008
88239f16fd40 8824e76a8000 8823e5fc2500 8823e5fc2500
88239f17 8845f5f070e8 8845f5f07100 8845f5f07080
0341bf10 88239f16fd60 b66a4d89 024000d00058
Call Trace:
[] schedule+0x39/0x80
[] rwsem_down_read_failed+0xc7/0x120
[] call_rwsem_down_read_failed+0x14/0x30
[] ? down_read+0x17/0x20
[] proc_pid_cmdline_read+0xac/0x4f0
[] ? acct_account_cputime+0x1c/0x20
[] ? account_user_time+0x73/0x80
[] ? vtime_account_user+0x4e/0x70
[] __vfs_read+0x18/0x40
[] vfs_read+0x8e/0x140
[] SyS_read+0x4f/0xa0
[] entry_SYSCALL_64_fastpath+0x12/0x71

[Qemu-devel] kernel 4.4.2: kvm_irq_delivery_to_api / rwsem_down_read_failed

[Stefan Priebe]

Hi,

while testing Kernel 4.4.2 and starting 20 Qemu 2.4.1 virtual machines. 
I got those traces and a load of 500 on those system. I was only abler 
to recover by sysrq-trigger.


All traces:

INFO: task pvedaemon worke:7470 blocked for more than 120 seconds.
Not tainted 4.4.2+1-ph #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pvedaemon worke D 88239c367ca0 0 7470 7468 0x0008
88239c367ca0 8840a6232500 8823ed83a500 88239c367c90
88239c368000 8845f5f070e8 8845f5f07100 
7ffc73b48e58 88239c367cc0 b66a4d89 88239c367cf0
Call Trace:
[] schedule+0x39/0x80
[] rwsem_down_read_failed+0xc7/0x120
[] call_rwsem_down_read_failed+0x14/0x30
[] ? down_read+0x17/0x20
[] __access_remote_vm+0x3e/0x1c0
[] ? call_rwsem_down_read_failed+0x14/0x30
[] access_remote_vm+0x1f/0x30
[] proc_pid_cmdline_read+0x16e/0x4f0
[] ? acct_account_cputime+0x1c/0x20
[] __vfs_read+0x18/0x40
[] vfs_read+0x8e/0x140
[] SyS_read+0x4f/0xa0
[] entry_SYSCALL_64_fastpath+0x12/0x71
INFO: task pvestatd:7633 blocked for more than 120 seconds.
Not tainted 4.4.2+1-ph #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pvestatd D 88239f16fd40 0 7633 1 0x0008
88239f16fd40 8824e76a8000 8823e5fc2500 8823e5fc2500
88239f17 8845f5f070e8 8845f5f07100 8845f5f07080
0341bf10 88239f16fd60 b66a4d89 024000d00058
Call Trace:
[] schedule+0x39/0x80
[] rwsem_down_read_failed+0xc7/0x120
[] call_rwsem_down_read_failed+0x14/0x30
[] ? down_read+0x17/0x20
[] proc_pid_cmdline_read+0xac/0x4f0
[] ? acct_account_cputime+0x1c/0x20
[] ? account_user_time+0x73/0x80
[] ? vtime_account_user+0x4e/0x70
[] __vfs_read+0x18/0x40
[] vfs_read+0x8e/0x140
[] SyS_read+0x4f/0xa0
[] entry_SYSCALL_64_fastpath+0x12/0x71
INFO: task kvm:11766 blocked for more than 120 seconds.
Not tainted 4.4.2+1-ph #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kvm D 88452a2d3dd0 0 11766 1 0x0008
88452a2d3dd0 880166c74a00 8845b7354a00 b617fc8e
88452a2d4000 8845f5f070e8 8845f5f07100 88452a2d3f58
8845b7354a00 88452a2d3df0 b66a4d89 7fa807abbf80
Call Trace:
[] ? __handle_mm_fault+0xd1e/0x1260
[] schedule+0x39/0x80
[] rwsem_down_read_failed+0xc7/0x120
[] call_rwsem_down_read_failed+0x14/0x30
[] ? down_read+0x17/0x20
[] __do_page_fault+0x2b7/0x380
[] ? account_user_time+0x73/0x80
[] ? vtime_account_user+0x4e/0x70
[] do_page_fault+0x37/0x90
[] page_fault+0x28/0x30
INFO: task kvm:11824 blocked for more than 120 seconds.
Not tainted 4.4.2+1-ph #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kvm D 8840a867faa0 0 11824 1 0x0008
8840a867faa0 8845866a4a00 8840a6232500 0001
8840a868 8845f5f070e8 8845f5f07100 8840a867fc0e
 8840a867fac0 b66a4d89 c0606a06
Call Trace:
[] schedule+0x39/0x80
[] ? kvm_irq_delivery_to_apic+0x56/0x220 [kvm]
[] rwsem_down_read_failed+0xc7/0x120
[] call_rwsem_down_read_failed+0x14/0x30
[] ? down_read+0x17/0x20
[] kvm_host_page_size+0x60/0xa0 [kvm]
[] mapping_level+0x5c/0x130 [kvm]
[] tdp_page_fault+0x9b/0x260 [kvm]
[] ? kernel_pio+0x2d/0x40 [kvm]
[] kvm_mmu_page_fault+0x31/0x120 [kvm]
[] handle_ept_violation+0xa4/0x170 [kvm_intel]
[] vmx_handle_exit+0x257/0x490 [kvm_intel]
[] ? __vtime_account_system+0x31/0x40
[] vcpu_enter_guest+0x6af/0xff0 [kvm]
[] ? kvm_apic_local_deliver+0x5d/0x60 [kvm]
[] kvm_arch_vcpu_ioctl_run+0xc4/0x3c0 [kvm]
[] kvm_vcpu_ioctl+0x324/0x5d0 [kvm]
[] ? acct_account_cputime+0x1c/0x20
[] ? account_user_time+0x73/0x80
[] do_vfs_ioctl+0x83/0x4e0
[] ? enter_from_user_mode+0x1f/0x50
[] ? syscall_trace_enter_phase1+0xc1/0x110
[] SyS_ioctl+0x4c/0x80
[] entry_SYSCALL_64_fastpath+0x12/0x71
INFO: task kvm:11825 blocked for more than 120 seconds.
Not tainted 4.4.2+1-ph #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kvm D 88458d6a3aa0 0 11825 1 0x00080002
88458d6a3aa0 880167302500 8840a6234a00 0001
88458d6a4000 8845f5f070e8 8845f5f07100 88458d6a3c0e
 88458d6a3ac0 b66a4d89 c0606a06
Call Trace:
[] schedule+0x39/0x80
[] ? kvm_irq_delivery_to_apic+0x56/0x220 [kvm]
[] rwsem_down_read_failed+0xc7/0x120
[] call_rwsem_down_read_failed+0x14/0x30
[] ? down_read+0x17/0x20
[] kvm_host_page_size+0x60/0xa0 [kvm]
[] mapping_level+0x5c/0x130 [kvm]
[] tdp_page_fault+0x9b/0x260 [kvm]
[] kvm_mmu_page_fault+0x31/0x120 [kvm]
[] handle_ept_violation+0xa4/0x170 [kvm_intel]
[] vmx_handle_exit+0x257/0x490 [kvm_intel]
[] ? __vtime_account_system+0x31/0x40
[] vcpu_enter_guest+0x6af/0xff0 [kvm]
[] ? kvm_apic_local_deliver+0x5d/0x60 [kvm]
[] kvm_arch_vcpu_ioctl_run+0xc4/0x3c0 [kvm]
[] kvm_vcpu_ioctl+0x324/0x5d0 [kvm]
[] ? acct_account_cputime+0x1c/0x20
[] ? account_user_time+0x73/0x80
[] do_vfs_ioctl+0x83/0x4e0
[] ? 

Re: [Qemu-devel] trace in arch/x86/kernel/apic/apic.c:1309 setup_local_APIC

[Stefan Priebe - Profihost AG]

Am 26.01.2016 um 11:13 schrieb Yang Zhang:
> On 2016/1/26 15:22, Stefan Priebe - Profihost AG wrote:
>> Hi,
>>
>> Am 26.01.2016 um 02:46 schrieb Han, Huaitong:
>>> What is the host kernel version and host dmesg information? And does
>>> the problem exist when you use latest kernel and QEMU to replace old
>>> binary file?
>>
>> Guest and Host is both 4.1.15. You mean the complete dmesg output from
>> host?
>>
>> What do you mean with replace old binary file? I haven't tested Kernel
>> 4.4 as we use 4.1 as it is a long term stable kernel release.
> 
> Have you seen this before? I mean use the old KVM like 3.10?

Guest or host? To test with a guest would be quite easy. Downgrading the
host is very difficult not sure if the hw is supported.

> 
>>
>> Stefan
>>
>>> On Mon, 2016-01-25 at 14:51 +0100, Stefan Priebe - Profihost AG wrote:
>>>> Hi,
>>>>
>>>> while running qemu 2.4 on whestmere CPUs i'm pretty often getting
>>>> this
>>>> one while booting:
>>>> [0.811645] Switched APIC routing to physical x2apic.
>>>> [1.835678] [ cut here ]
>>>> [1.835704] WARNING: CPU: 0 PID: 1 at
>>>> arch/x86/kernel/apic/apic.c:1309 setup_local_APIC+0x284/0x340()
>>>> [1.835714] Modules linked in:
>>>> [1.835724] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.15+72-ph
>>>> #1
>>>> [1.835731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>>>> BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
>>>> [1.835743]  b69ffcea 88042d5e3d68 b669c37b
>>>> 0918
>>>> [1.835754]   88042d5e3da8 b6080d67
>>>> 88042d5e3da8
>>>> [1.835765]  0001  8000
>>>> 
>>>> [1.835777] Call Trace:
>>>> [1.835789]  [] dump_stack+0x45/0x57
>>>> [1.835799]  [] warn_slowpath_common+0x97/0xe0
>>>> [1.835806]  [] warn_slowpath_null+0x1a/0x20
>>>> [1.835813]  [] setup_local_APIC+0x284/0x340
>>>> [1.835824]  [] apic_bsp_setup+0x5b/0xb0
>>>> [1.835832]  []
>>>> native_smp_prepare_cpus+0x23b/0x295
>>>> [1.835842]  [] kernel_init_freeable+0xc7/0x20f
>>>> [1.835853]  [] ? rest_init+0x80/0x80
>>>> [1.835860]  [] kernel_init+0xe/0xf0
>>>> [1.835870]  [] ret_from_fork+0x42/0x70
>>>> [1.835877]  [] ? rest_init+0x80/0x80
>>>> [1.835891] ---[ end trace bdbe630a8de2832c ]---
>>>> [1.837613] Spurious LAPIC timer interrupt on cpu 0
>>>> [1.837957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>>>> [1.939574] smpboot: CPU0: Intel Westmere E56xx/L56xx/X56xx
>>>> (Nehalem-C) (fam: 06, model: 2c, stepping: 01)
>>>> [1.939630] Performance Events: unsupported p6 CPU model 44 no PMU
>>>> driver, software events only.
>>>> [1.950868] KVM setup paravirtual spinlock
>>>>
>>>> Greets,
>>>> Stefan
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majord...@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> 

Re: [Qemu-devel] trace in arch/x86/kernel/apic/apic.c:1309 setup_local_APIC

[Stefan Priebe - Profihost AG]

Am 26.01.2016 um 12:39 schrieb Yang Zhang:
> On 2016/1/26 18:43, Stefan Priebe - Profihost AG wrote:
>>
>> Am 26.01.2016 um 11:13 schrieb Yang Zhang:
>>> On 2016/1/26 15:22, Stefan Priebe - Profihost AG wrote:
>>>> Hi,
>>>>
>>>> Am 26.01.2016 um 02:46 schrieb Han, Huaitong:
>>>>> What is the host kernel version and host dmesg information? And does
>>>>> the problem exist when you use latest kernel and QEMU to replace old
>>>>> binary file?
>>>>
>>>> Guest and Host is both 4.1.15. You mean the complete dmesg output from
>>>> host?
>>>>
>>>> What do you mean with replace old binary file? I haven't tested Kernel
>>>> 4.4 as we use 4.1 as it is a long term stable kernel release.
>>>
>>> Have you seen this before? I mean use the old KVM like 3.10?
>>
>> Guest or host? To test with a guest would be quite easy. Downgrading the
>> host is very difficult not sure if the hw is supported.
> 
> Host. Does the issue only exist on the Westmere CPU?

Yes. All E5 Xeons v1, v2, v3 are working fine and i've never seen this
message.

Stefan

> 
>>
>>>
>>>>
>>>> Stefan
>>>>
>>>>> On Mon, 2016-01-25 at 14:51 +0100, Stefan Priebe - Profihost AG wrote:
>>>>>> Hi,
>>>>>>
>>>>>> while running qemu 2.4 on whestmere CPUs i'm pretty often getting
>>>>>> this
>>>>>> one while booting:
>>>>>> [0.811645] Switched APIC routing to physical x2apic.
>>>>>> [1.835678] [ cut here ]
>>>>>> [1.835704] WARNING: CPU: 0 PID: 1 at
>>>>>> arch/x86/kernel/apic/apic.c:1309 setup_local_APIC+0x284/0x340()
>>>>>> [1.835714] Modules linked in:
>>>>>> [1.835724] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.15+72-ph
>>>>>> #1
>>>>>> [1.835731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>>>>>> BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
>>>>>> [1.835743]  b69ffcea 88042d5e3d68 b669c37b
>>>>>> 0918
>>>>>> [1.835754]   88042d5e3da8 b6080d67
>>>>>> 88042d5e3da8
>>>>>> [1.835765]  0001  8000
>>>>>> 
>>>>>> [1.835777] Call Trace:
>>>>>> [1.835789]  [] dump_stack+0x45/0x57
>>>>>> [1.835799]  [] warn_slowpath_common+0x97/0xe0
>>>>>> [1.835806]  [] warn_slowpath_null+0x1a/0x20
>>>>>> [1.835813]  [] setup_local_APIC+0x284/0x340
>>>>>> [1.835824]  [] apic_bsp_setup+0x5b/0xb0
>>>>>> [1.835832]  []
>>>>>> native_smp_prepare_cpus+0x23b/0x295
>>>>>> [1.835842]  [] kernel_init_freeable+0xc7/0x20f
>>>>>> [1.835853]  [] ? rest_init+0x80/0x80
>>>>>> [1.835860]  [] kernel_init+0xe/0xf0
>>>>>> [1.835870]  [] ret_from_fork+0x42/0x70
>>>>>> [1.835877]  [] ? rest_init+0x80/0x80
>>>>>> [1.835891] ---[ end trace bdbe630a8de2832c ]---
>>>>>> [1.837613] Spurious LAPIC timer interrupt on cpu 0
>>>>>> [1.837957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>>>>>> [1.939574] smpboot: CPU0: Intel Westmere E56xx/L56xx/X56xx
>>>>>> (Nehalem-C) (fam: 06, model: 2c, stepping: 01)
>>>>>> [1.939630] Performance Events: unsupported p6 CPU model 44 no PMU
>>>>>> driver, software events only.
>>>>>> [1.950868] KVM setup paravirtual spinlock
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>> -- 
>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>> the body of a message to majord...@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majord...@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>>>
> 
> 

Re: [Qemu-devel] trace in arch/x86/kernel/apic/apic.c:1309 setup_local_APIC

[Stefan Priebe - Profihost AG]

Hi,

Am 26.01.2016 um 02:46 schrieb Han, Huaitong:
> What is the host kernel version and host dmesg information? And does
> the problem exist when you use latest kernel and QEMU to replace old
> binary file?

Guest and Host is both 4.1.15. You mean the complete dmesg output from host?

What do you mean with replace old binary file? I haven't tested Kernel
4.4 as we use 4.1 as it is a long term stable kernel release.

Stefan

> On Mon, 2016-01-25 at 14:51 +0100, Stefan Priebe - Profihost AG wrote:
>> Hi,
>>
>> while running qemu 2.4 on whestmere CPUs i'm pretty often getting
>> this
>> one while booting:
>> [0.811645] Switched APIC routing to physical x2apic.
>> [1.835678] [ cut here ]
>> [1.835704] WARNING: CPU: 0 PID: 1 at
>> arch/x86/kernel/apic/apic.c:1309 setup_local_APIC+0x284/0x340()
>> [1.835714] Modules linked in:
>> [1.835724] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.15+72-ph
>> #1
>> [1.835731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>> BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
>> [1.835743]  b69ffcea 88042d5e3d68 b669c37b
>> 0918
>> [1.835754]   88042d5e3da8 b6080d67
>> 88042d5e3da8
>> [1.835765]  0001  8000
>> 
>> [1.835777] Call Trace:
>> [1.835789]  [] dump_stack+0x45/0x57
>> [1.835799]  [] warn_slowpath_common+0x97/0xe0
>> [1.835806]  [] warn_slowpath_null+0x1a/0x20
>> [1.835813]  [] setup_local_APIC+0x284/0x340
>> [1.835824]  [] apic_bsp_setup+0x5b/0xb0
>> [1.835832]  []
>> native_smp_prepare_cpus+0x23b/0x295
>> [1.835842]  [] kernel_init_freeable+0xc7/0x20f
>> [1.835853]  [] ? rest_init+0x80/0x80
>> [1.835860]  [] kernel_init+0xe/0xf0
>> [1.835870]  [] ret_from_fork+0x42/0x70
>> [1.835877]  [] ? rest_init+0x80/0x80
>> [1.835891] ---[ end trace bdbe630a8de2832c ]---
>> [1.837613] Spurious LAPIC timer interrupt on cpu 0
>> [1.837957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>> [1.939574] smpboot: CPU0: Intel Westmere E56xx/L56xx/X56xx
>> (Nehalem-C) (fam: 06, model: 2c, stepping: 01)
>> [1.939630] Performance Events: unsupported p6 CPU model 44 no PMU
>> driver, software events only.
>> [1.950868] KVM setup paravirtual spinlock
>>
>> Greets,
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Qemu-devel] trace in arch/x86/kernel/apic/apic.c:1309 setup_local_APIC

[Stefan Priebe - Profihost AG]

Hi,

while running qemu 2.4 on whestmere CPUs i'm pretty often getting this
one while booting:
[0.811645] Switched APIC routing to physical x2apic.
[1.835678] [ cut here ]
[1.835704] WARNING: CPU: 0 PID: 1 at
arch/x86/kernel/apic/apic.c:1309 setup_local_APIC+0x284/0x340()
[1.835714] Modules linked in:
[1.835724] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.15+72-ph #1
[1.835731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
[1.835743]  b69ffcea 88042d5e3d68 b669c37b
0918
[1.835754]   88042d5e3da8 b6080d67
88042d5e3da8
[1.835765]  0001  8000

[1.835777] Call Trace:
[1.835789]  [] dump_stack+0x45/0x57
[1.835799]  [] warn_slowpath_common+0x97/0xe0
[1.835806]  [] warn_slowpath_null+0x1a/0x20
[1.835813]  [] setup_local_APIC+0x284/0x340
[1.835824]  [] apic_bsp_setup+0x5b/0xb0
[1.835832]  [] native_smp_prepare_cpus+0x23b/0x295
[1.835842]  [] kernel_init_freeable+0xc7/0x20f
[1.835853]  [] ? rest_init+0x80/0x80
[1.835860]  [] kernel_init+0xe/0xf0
[1.835870]  [] ret_from_fork+0x42/0x70
[1.835877]  [] ? rest_init+0x80/0x80
[1.835891] ---[ end trace bdbe630a8de2832c ]---
[1.837613] Spurious LAPIC timer interrupt on cpu 0
[1.837957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[1.939574] smpboot: CPU0: Intel Westmere E56xx/L56xx/X56xx
(Nehalem-C) (fam: 06, model: 2c, stepping: 01)
[1.939630] Performance Events: unsupported p6 CPU model 44 no PMU
driver, software events only.
[1.950868] KVM setup paravirtual spinlock

Greets,
Stefan

Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop is hanging forever

[Stefan Priebe - Profihost AG]

> - Original Message -
>> From: "Alexandre DERUMIER" 
>> To: "ceph-devel" 
>> Cc: "qemu-devel" , jdur...@redhat.com
>> Sent: Monday, November 9, 2015 5:48:45 AM
>> Subject: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and 
>> vm_stop is hanging forever
>>
>> adding to ceph.conf
>>
>> [client]
>> rbd_non_blocking_aio = false
>>
>>
>> fix the problem for me (with rbd_cache=false)
>>
>>
>> (@cc jdur...@redhat.com)

+1 same to me.

Stefan

>>
>>
>>
>> - Mail original -
>> De: "Denis V. Lunev" 
>> À: "aderumier" , "ceph-devel"
>> , "qemu-devel" 
>> Envoyé: Lundi 9 Novembre 2015 08:22:34
>> Objet: Re: [Qemu-devel] qemu : rbd block driver internal snapshot and vm_stop
>> is hanging forever
>>
>> On 11/09/2015 10:19 AM, Denis V. Lunev wrote:
>>> On 11/09/2015 06:10 AM, Alexandre DERUMIER wrote:
 Hi,

 with qemu (2.4.1), if I do an internal snapshot of an rbd device,
 then I pause the vm with vm_stop,

 the qemu process is hanging forever


 monitor commands to reproduce:


 # snapshot_blkdev_internal drive-virtio0 yoursnapname
 # stop




 I don't see this with qcow2 or sheepdog block driver for example.


 Regards,

 Alexandre

>>> this could look like the problem I have recenty trying to
>>> fix with dataplane enabled. Patch series is named as
>>>
>>> [PATCH for 2.5 v6 0/10] dataplane snapshot fixes
>>>
>>> Den
>>
>> anyway, even if above will not help, can you collect gdb
>> traces from all threads in QEMU process. May be I'll be
>> able to give a hit.
>>
>> Den
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [Qemu-devel] guest: NO_HZ FULL will not work with unstable sched clock

[Stefan Priebe]

same with kvm clock:

[0.195073] Switched to clocksource kvm-clock
[0.195509] [ cut here ]
[0.195509] WARNING: CPU: 1 PID: 1 at kernel/time/tick-sched.c:192 
can_stop_full_tick+0x1c1/0x200()

[0.195515] NO_HZ FULL will not work with unstable sched clock
[0.195518] Modules linked in:
[0.195531] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.1.5+11-ph #1
[0.195535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org 04/01/2014
[0.195544]  819fc200 88023fc43e88 8162dd36 
121a
[0.195551]  88023fc43ed8 88023fc43ec8 8107dcb7 
88023fc43ea8
[0.195558]   0001 81e667c0 
81cf1540

[0.195564] Call Trace:
[0.195568]  IRQ  [8162dd36] dump_stack+0x45/0x57
[0.195591]  [8107dcb7] warn_slowpath_common+0x97/0xe0
[0.195596]  [8107ddb6] warn_slowpath_fmt+0x46/0x50
[0.195600]  [810f08c1] can_stop_full_tick+0x1c1/0x200
[0.195604]  [810f0c87] tick_nohz_irq_exit+0x97/0xd0
[0.195609]  [81081c3d] irq_exit+0xcd/0x130
[0.195620]  [8103aeaa] smp_apic_timer_interrupt+0x4a/0x60
[0.195627]  [816351fe] apic_timer_interrupt+0x6e/0x80
[0.195631]  EOI  [81132e87] ? 
trace_event_enum_update+0x1c7/0x3b0

[0.195644]  [81132ecc] ? trace_event_enum_update+0x20c/0x3b0
[0.195649]  [81d77d01] ? stop_trace_on_warning+0x3f/0x3f
[0.195654]  [81d77e1b] tracer_init_tracefs+0x11a/0x23b
[0.195661]  [81000328] do_one_initcall+0xd8/0x210
[0.195669]  [8109b700] ? parse_args+0x1b0/0x450
[0.195676]  [810c0d23] ? __wake_up+0x53/0x70
[0.195684]  [81d5a265] kernel_init_freeable+0x178/0x20f
[0.195689]  [81d5997d] ? do_early_param+0x8c/0x8c
[0.195700]  [811c76c2] ? create_pipe_files+0x1f2/0x200
[0.195706]  [8161f6a0] ? rest_init+0x80/0x80
[0.195711]  [8161f6ae] kernel_init+0xe/0xf0
[0.195715]  [816347a2] ret_from_fork+0x42/0x70
[0.195719]  [8161f6a0] ? rest_init+0x80/0x80
[0.195729] ---[ end trace cf665146248feec1 ]---

Stefan

Am 15.08.2015 um 20:44 schrieb Stefan Priebe:

Hi,

while switching to a FULL tickless kernel i detected that all our VMs
produce the following stack trace while running under qemu 2.3.0.

[0.195160] HPET: 3 timers in total, 0 timers will be used for
per-cpu timer
[0.195181] hpet0: at MMIO 0xfed0, IRQs 2, 8, 0
[0.195188] hpet0: 3 comparators, 64-bit 100.00 MHz counter
[0.200119] Switched to clocksource hpet
[0.201164] [ cut here ]
[0.201193] WARNING: CPU: 1 PID: 1 at kernel/time/tick-sched.c:192
can_stop_full_tick+0x1c1/0x200()
[0.201199] NO_HZ FULL will not work with unstable sched clock
[0.201203] Modules linked in:
[0.201214] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.18.20+55-ph #1
[0.201219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org
04/01/2014
[0.201226]  0009 88023fc43e88 815f5889
08df
[0.201230]  88023fc43ed8 88023fc43ec8 8107433c
88023fc43ea8
[0.201233]   0001 880236c41300
cbbde78b
[0.201237] Call Trace:
[0.201240]  IRQ  [815f5889] dump_stack+0x46/0x58
[0.201260]  [8107433c] warn_slowpath_common+0x8c/0xc0
[0.201264]  [81074426] warn_slowpath_fmt+0x46/0x50
[0.201270]  [810ddc31] can_stop_full_tick+0x1c1/0x200
[0.201274]  [810ddfc7] tick_nohz_irq_exit+0x97/0xd0
[0.201277]  [81077f0d] irq_exit+0xcd/0x130
[0.201286]  [8103419a] smp_apic_timer_interrupt+0x4a/0x60
[0.201293]  [815fc4ea] apic_timer_interrupt+0x6a/0x70
[0.201296]  EOI  [811bc1d5] ? __d_lookup+0x55/0x160
[0.201309]  [811bc315] d_lookup+0x35/0x60
[0.201312]  [811adc1b] lookup_dcache+0x3b/0xd0
[0.201315]  [811adcd3] __lookup_hash+0x23/0x50
[0.201319]  [811aeaae] lookup_one_len+0xce/0x120
[0.201328]  [812d062f] __create_file+0x8f/0x260
[0.201332]  [812d089b] debugfs_create_dir+0x1b/0x20
[0.201338]  [8111d9a8] event_create_dir+0x68/0x520
[0.201342]  [81d5bdb0] event_trace_init+0x188/0x210
[0.201348]  [81113e1b] ? trace_create_file+0x1b/0x50
[0.201352]  [81d5bc28] ? event_trace_enable+0x1a2/0x1a2
[0.201359]  [81000328] do_one_initcall+0xd8/0x200
[0.201368]  [81090200] ? parse_args+0x1f0/0x470
[0.201373]  [810b1c33] ? __wake_up+0x53/0x70
[0.201378]  [81d3e1ce] kernel_init_freeable+0x168/0x1ff
[0.201381]  [81d3d917] ? do_early_param+0x8c/0x8c

[Qemu-devel] guest: NO_HZ FULL will not work with unstable sched clock

[Stefan Priebe]

Hi,

while switching to a FULL tickless kernel i detected that all our VMs
produce the following stack trace while running under qemu 2.3.0.

[0.195160] HPET: 3 timers in total, 0 timers will be used for
per-cpu timer
[0.195181] hpet0: at MMIO 0xfed0, IRQs 2, 8, 0
[0.195188] hpet0: 3 comparators, 64-bit 100.00 MHz counter
[0.200119] Switched to clocksource hpet
[0.201164] [ cut here ]
[0.201193] WARNING: CPU: 1 PID: 1 at kernel/time/tick-sched.c:192
can_stop_full_tick+0x1c1/0x200()
[0.201199] NO_HZ FULL will not work with unstable sched clock
[0.201203] Modules linked in:
[0.201214] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.18.20+55-ph #1
[0.201219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS rel-1.8.1-0-g4adadbd-20150316_085822-nilsson.home.kraxel.org 04/01/2014
[0.201226]  0009 88023fc43e88 815f5889
08df
[0.201230]  88023fc43ed8 88023fc43ec8 8107433c
88023fc43ea8
[0.201233]   0001 880236c41300
cbbde78b
[0.201237] Call Trace:
[0.201240]  IRQ  [815f5889] dump_stack+0x46/0x58
[0.201260]  [8107433c] warn_slowpath_common+0x8c/0xc0
[0.201264]  [81074426] warn_slowpath_fmt+0x46/0x50
[0.201270]  [810ddc31] can_stop_full_tick+0x1c1/0x200
[0.201274]  [810ddfc7] tick_nohz_irq_exit+0x97/0xd0
[0.201277]  [81077f0d] irq_exit+0xcd/0x130
[0.201286]  [8103419a] smp_apic_timer_interrupt+0x4a/0x60
[0.201293]  [815fc4ea] apic_timer_interrupt+0x6a/0x70
[0.201296]  EOI  [811bc1d5] ? __d_lookup+0x55/0x160
[0.201309]  [811bc315] d_lookup+0x35/0x60
[0.201312]  [811adc1b] lookup_dcache+0x3b/0xd0
[0.201315]  [811adcd3] __lookup_hash+0x23/0x50
[0.201319]  [811aeaae] lookup_one_len+0xce/0x120
[0.201328]  [812d062f] __create_file+0x8f/0x260
[0.201332]  [812d089b] debugfs_create_dir+0x1b/0x20
[0.201338]  [8111d9a8] event_create_dir+0x68/0x520
[0.201342]  [81d5bdb0] event_trace_init+0x188/0x210
[0.201348]  [81113e1b] ? trace_create_file+0x1b/0x50
[0.201352]  [81d5bc28] ? event_trace_enable+0x1a2/0x1a2
[0.201359]  [81000328] do_one_initcall+0xd8/0x200
[0.201368]  [81090200] ? parse_args+0x1f0/0x470
[0.201373]  [810b1c33] ? __wake_up+0x53/0x70
[0.201378]  [81d3e1ce] kernel_init_freeable+0x168/0x1ff
[0.201381]  [81d3d917] ? do_early_param+0x8c/0x8c
[0.201389]  [812f0680] ? cryptomgr_probe+0x60/0xe0
[0.201398]  [815e6f30] ? rest_init+0x80/0x80
[0.201401]  [815e6f3e] kernel_init+0xe/0xf0
[0.201405]  [815fb708] ret_from_fork+0x58/0x90
[0.201408]  [815e6f30] ? rest_init+0x80/0x80
[0.201415] ---[ end trace a15ba3bc1d247b4c ]---

Stefan

Re: [Qemu-devel] [Qemu-stable] [PULL 0/3] Cve 2015 5154 patches

[Stefan Priebe - Profihost AG]

Am 27.07.2015 um 14:01 schrieb John Snow:
 The following changes since commit f793d97e454a56d17e404004867985622ca1a63b:
 
   Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into 
 staging (2015-07-24 13:07:10 +0100)
 
 are available in the git repository at:
 
   https://github.com/jnsnow/qemu.git tags/cve-2015-5154-pull-request

Any details on this CVE? Is RCE possible? Only if IDE is used?

Stefan

 for you to fetch changes up to cb72cba83021fa42719e73a5249c12096a4d1cfc:
 
   ide: Clear DRQ after handling all expected accesses (2015-07-26 23:42:53 
 -0400)
 
 
 
 
 
 Kevin Wolf (3):
   ide: Check array bounds before writing to io_buffer (CVE-2015-5154)
   ide/atapi: Fix START STOP UNIT command completion
   ide: Clear DRQ after handling all expected accesses
 
  hw/ide/atapi.c |  1 +
  hw/ide/core.c  | 32 
  2 files changed, 29 insertions(+), 4 deletions(-)
 

Re: [Qemu-devel] [Qemu-stable] [PULL 0/3] Cve 2015 5154 patches

[Stefan Priebe - Profihost AG]

Am 27.07.2015 um 14:28 schrieb John Snow:
 
 
 On 07/27/2015 08:10 AM, Stefan Priebe - Profihost AG wrote:

 Am 27.07.2015 um 14:01 schrieb John Snow:
 The following changes since commit f793d97e454a56d17e404004867985622ca1a63b:

   Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into 
 staging (2015-07-24 13:07:10 +0100)

 are available in the git repository at:

   https://github.com/jnsnow/qemu.git tags/cve-2015-5154-pull-request

 Any details on this CVE? Is RCE possible? Only if IDE is used?

 Stefan

 
 It's a heap overflow. The most likely outcome is a segfault, but the
 guest is allowed to continue writing past the end of the PIO buffer at
 its leisure. This makes it similar to CVE-2015-3456.
 
 This CVE can be mitigated unlike CVE-2015-3456 by just removing the
 CD-ROM drive until the patch can be applied.

Thanks. The seclist article explicitly references xen. So it does not
apply to qemu/kvm? Sorry for asking may be stupid questions.

Stefan

 for you to fetch changes up to cb72cba83021fa42719e73a5249c12096a4d1cfc:

   ide: Clear DRQ after handling all expected accesses (2015-07-26 23:42:53 
 -0400)

 

 

 Kevin Wolf (3):
   ide: Check array bounds before writing to io_buffer (CVE-2015-5154)
   ide/atapi: Fix START STOP UNIT command completion
   ide: Clear DRQ after handling all expected accesses

  hw/ide/atapi.c |  1 +
  hw/ide/core.c  | 32 
  2 files changed, 29 insertions(+), 4 deletions(-)

[Qemu-devel] Query CPU model / type

[Stefan Priebe - Profihost AG]

Hi,

is there a way to query the current cpu model / type of a running qemu
machine?

I mean host, kvm64, qemu64, ...

Stefan

Re: [Qemu-devel] Query CPU model / type

[Stefan Priebe]

Am 15.07.2015 um 13:32 schrieb Andrey Korolyov:

On Wed, Jul 15, 2015 at 2:20 PM, Stefan Priebe - Profihost AG
s.pri...@profihost.ag wrote:

Hi,

is there a way to query the current cpu model / type of a running qemu
machine?

I mean host, kvm64, qemu64, ...

Stefan



I believe that the most proper one would be
'query-command-line-options'. The CPU definitions are not exposed
anywhere else in QMP except their list.


But this one only lists the possible arguments not the currently passed 
ones?


Stefan

Re: [Qemu-devel] Query CPU model / type

[Stefan Priebe]

Am 15.07.2015 um 22:15 schrieb Andrey Korolyov:

On Wed, Jul 15, 2015 at 11:07 PM, Stefan Priebe s.pri...@profihost.ag wrote:

Am 15.07.2015 um 13:32 schrieb Andrey Korolyov:


On Wed, Jul 15, 2015 at 2:20 PM, Stefan Priebe - Profihost AG
s.pri...@profihost.ag wrote:


Hi,

is there a way to query the current cpu model / type of a running qemu
machine?

I mean host, kvm64, qemu64, ...

Stefan



I believe that the most proper one would be
'query-command-line-options'. The CPU definitions are not exposed
anywhere else in QMP except their list.



But this one only lists the possible arguments not the currently passed
ones?

Stefan


Sorry, my bad, wrong memories. So there is no way to retrieve
arguments via monitor.


and even no current cpu model ;-( pretty bad for migrations. So i'll 
have to hack the qemu code.


Stefan

Re: [Qemu-devel] [Qemu-stable] [PATCH] fdc: force the fifo access to be in bounds of the allocated buffer

[Stefan Priebe]

Am 13.05.2015 um 21:05 schrieb Stefan Weil:

Am 13.05.2015 um 20:59 schrieb Stefan Priebe:


Am 13.05.2015 um 20:51 schrieb Stefan Weil:

Hi,

I just noticed this patch because my provider told me that my KVM based
server
needs a reboot because of a CVE (see this German news:
http://www.heise.de/newsticker/meldung/Venom-Schwachstelle-Aus-Hypervisor-ausbrechen-und-VMs-ausspionieren-2649614.html)



Isn't a live migration to a fixed version enough instead of a reboot?

Stefan


Good point. A live migration would be sufficient - if there are no bugs
in QEMU's live migration.


just migrating all our customer machines and wanted to be sure that live 
migration is enough.


Greets,
Stefan



Stefan

Re: [Qemu-devel] [Qemu-stable] [PATCH] fdc: force the fifo access to be in bounds of the allocated buffer

[Stefan Priebe]

Am 13.05.2015 um 20:51 schrieb Stefan Weil:

Hi,

I just noticed this patch because my provider told me that my KVM based
server
needs a reboot because of a CVE (see this German news:
http://www.heise.de/newsticker/meldung/Venom-Schwachstelle-Aus-Hypervisor-ausbrechen-und-VMs-ausspionieren-2649614.html)


Isn't a live migration to a fixed version enough instead of a reboot?

Stefan




Am 13.05.2015 um 16:33 schrieb John Snow:

From: Petr Matousek pmato...@redhat.com

During processing of certain commands such as FD_CMD_READ_ID and
FD_CMD_DRIVE_SPECIFICATION_COMMAND the fifo memory access could
get out of bounds leading to memory corruption with values coming
from the guest.

Fix this by making sure that the index is always bounded by the
allocated memory.

This is CVE-2015-3456.

Signed-off-by: Petr Matousek pmato...@redhat.com
Reviewed-by: John Snow js...@redhat.com
Signed-off-by: John Snow js...@redhat.com
---
  hw/block/fdc.c | 17 +++--
  1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/hw/block/fdc.c b/hw/block/fdc.c
index f72a392..d8a8edd 100644
--- a/hw/block/fdc.c
+++ b/hw/block/fdc.c
@@ -1497,7 +1497,7 @@ static uint32_t fdctrl_read_data(FDCtrl *fdctrl)
  {
  FDrive *cur_drv;
  uint32_t retval = 0;
-int pos;
+uint32_t pos;
  cur_drv = get_cur_drv(fdctrl);
  fdctrl-dsr = ~FD_DSR_PWRDOWN;
@@ -1506,8 +1506,8 @@ static uint32_t fdctrl_read_data(FDCtrl *fdctrl)
  return 0;
  }
  pos = fdctrl-data_pos;
+pos %= FD_SECTOR_LEN;


I'd combine both statements and perhaps use fdctrl-fifo_size (even if
the resulting code will be slightly larger):

pos = fdctrl-data_pos % fdctrl-fifo_size;



  if (fdctrl-msr  FD_MSR_NONDMA) {
-pos %= FD_SECTOR_LEN;
  if (pos == 0) {
  if (fdctrl-data_pos != 0)
  if (!fdctrl_seek_to_next_sect(fdctrl, cur_drv)) {
@@ -1852,10 +1852,13 @@ static void fdctrl_handle_option(FDCtrl
*fdctrl, int direction)
  static void fdctrl_handle_drive_specification_command(FDCtrl
*fdctrl, int direction)
  {
  FDrive *cur_drv = get_cur_drv(fdctrl);
+uint32_t pos;
-if (fdctrl-fifo[fdctrl-data_pos - 1]  0x80) {
+pos = fdctrl-data_pos - 1;
+pos %= FD_SECTOR_LEN;


Shorter (and more clear):

uint32_t pos = (fdctrl-data_pos - 1) % fdctrl-fifo_size;


+if (fdctrl-fifo[pos]  0x80) {
  /* Command parameters done */
-if (fdctrl-fifo[fdctrl-data_pos - 1]  0x40) {
+if (fdctrl-fifo[pos]  0x40) {
  fdctrl-fifo[0] = fdctrl-fifo[1];
  fdctrl-fifo[2] = 0;
  fdctrl-fifo[3] = 0;
@@ -1955,7 +1958,7 @@ static uint8_t command_to_handler[256];
  static void fdctrl_write_data(FDCtrl *fdctrl, uint32_t value)
  {
  FDrive *cur_drv;
-int pos;
+uint32_t pos;
  /* Reset mode */
  if (!(fdctrl-dor  FD_DOR_nRESET)) {
@@ -2004,7 +2007,9 @@ static void fdctrl_write_data(FDCtrl *fdctrl,
uint32_t value)
  }
  FLOPPY_DPRINTF(%s: %02x\n, __func__, value);
-fdctrl-fifo[fdctrl-data_pos++] = value;
+pos = fdctrl-data_pos++;
+pos %= FD_SECTOR_LEN;
+fdctrl-fifo[pos] = value;
  if (fdctrl-data_pos == fdctrl-data_len) {
  /* We now have all parameters
   * and will be able to treat the command


Not strictly related to this patch: The code which sets fifo_size could
also be improved.

 fdctrl-fifo = qemu_memalign(512, FD_SECTOR_LEN);
 fdctrl-fifo_size = 512;

The 2nd line should be

 fdctrl-fifo_size = FD_SECTOR_LEN;


As far as I see the original code can read or write illegal memory
locations in the address space of the QEMU process. It cannot (as it was
claimed) modify the code of the VM host because those memory is usually
write protected - at least if QEMU is running without KVM. If the code
which is generated for KVM is writable from anywhere in QEMU, we should
perhaps consider changing that.

Regards
Stefan

Re: [Qemu-devel] [Qemu-stable] [PATCH] fdc: force the fifo access to be in bounds of the allocated buffer

[Stefan Priebe]

Am 13.05.2015 um 21:04 schrieb John Snow:



On 05/13/2015 02:59 PM, Stefan Priebe wrote:


Am 13.05.2015 um 20:51 schrieb Stefan Weil:

Hi,

I just noticed this patch because my provider told me that my KVM based
server
needs a reboot because of a CVE (see this German news:
http://www.heise.de/newsticker/meldung/Venom-Schwachstelle-Aus-Hypervisor-ausbrechen-und-VMs-ausspionieren-2649614.html)



Isn't a live migration to a fixed version enough instead of a reboot?

Stefan




If your management API or host or whatever lets you migrate back to the
same host, or has another host they can migrate you to, yes.


i'm my host ;-) I was just interested if for whatever reason live 
migration is not enough and was wondering why someone wants to reboot a 
vm for something like this.


Thanks!

Stefan





Am 13.05.2015 um 16:33 schrieb John Snow:

From: Petr Matousek pmato...@redhat.com

During processing of certain commands such as FD_CMD_READ_ID and
FD_CMD_DRIVE_SPECIFICATION_COMMAND the fifo memory access could
get out of bounds leading to memory corruption with values coming
from the guest.

Fix this by making sure that the index is always bounded by the
allocated memory.

This is CVE-2015-3456.

Signed-off-by: Petr Matousek pmato...@redhat.com
Reviewed-by: John Snow js...@redhat.com
Signed-off-by: John Snow js...@redhat.com
---
   hw/block/fdc.c | 17 +++--
   1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/hw/block/fdc.c b/hw/block/fdc.c
index f72a392..d8a8edd 100644
--- a/hw/block/fdc.c
+++ b/hw/block/fdc.c
@@ -1497,7 +1497,7 @@ static uint32_t fdctrl_read_data(FDCtrl *fdctrl)
   {
   FDrive *cur_drv;
   uint32_t retval = 0;
-int pos;
+uint32_t pos;
   cur_drv = get_cur_drv(fdctrl);
   fdctrl-dsr = ~FD_DSR_PWRDOWN;
@@ -1506,8 +1506,8 @@ static uint32_t fdctrl_read_data(FDCtrl *fdctrl)
   return 0;
   }
   pos = fdctrl-data_pos;
+pos %= FD_SECTOR_LEN;


I'd combine both statements and perhaps use fdctrl-fifo_size (even if
the resulting code will be slightly larger):

pos = fdctrl-data_pos % fdctrl-fifo_size;



   if (fdctrl-msr  FD_MSR_NONDMA) {
-pos %= FD_SECTOR_LEN;
   if (pos == 0) {
   if (fdctrl-data_pos != 0)
   if (!fdctrl_seek_to_next_sect(fdctrl, cur_drv)) {
@@ -1852,10 +1852,13 @@ static void fdctrl_handle_option(FDCtrl
*fdctrl, int direction)
   static void fdctrl_handle_drive_specification_command(FDCtrl
*fdctrl, int direction)
   {
   FDrive *cur_drv = get_cur_drv(fdctrl);
+uint32_t pos;
-if (fdctrl-fifo[fdctrl-data_pos - 1]  0x80) {
+pos = fdctrl-data_pos - 1;
+pos %= FD_SECTOR_LEN;


Shorter (and more clear):

uint32_t pos = (fdctrl-data_pos - 1) % fdctrl-fifo_size;


+if (fdctrl-fifo[pos]  0x80) {
   /* Command parameters done */
-if (fdctrl-fifo[fdctrl-data_pos - 1]  0x40) {
+if (fdctrl-fifo[pos]  0x40) {
   fdctrl-fifo[0] = fdctrl-fifo[1];
   fdctrl-fifo[2] = 0;
   fdctrl-fifo[3] = 0;
@@ -1955,7 +1958,7 @@ static uint8_t command_to_handler[256];
   static void fdctrl_write_data(FDCtrl *fdctrl, uint32_t value)
   {
   FDrive *cur_drv;
-int pos;
+uint32_t pos;
   /* Reset mode */
   if (!(fdctrl-dor  FD_DOR_nRESET)) {
@@ -2004,7 +2007,9 @@ static void fdctrl_write_data(FDCtrl *fdctrl,
uint32_t value)
   }
   FLOPPY_DPRINTF(%s: %02x\n, __func__, value);
-fdctrl-fifo[fdctrl-data_pos++] = value;
+pos = fdctrl-data_pos++;
+pos %= FD_SECTOR_LEN;
+fdctrl-fifo[pos] = value;
   if (fdctrl-data_pos == fdctrl-data_len) {
   /* We now have all parameters
* and will be able to treat the command


Not strictly related to this patch: The code which sets fifo_size could
also be improved.

  fdctrl-fifo = qemu_memalign(512, FD_SECTOR_LEN);
  fdctrl-fifo_size = 512;

The 2nd line should be

  fdctrl-fifo_size = FD_SECTOR_LEN;


As far as I see the original code can read or write illegal memory
locations in the address space of the QEMU process. It cannot (as it was
claimed) modify the code of the VM host because those memory is usually
write protected - at least if QEMU is running without KVM. If the code
which is generated for KVM is writable from anywhere in QEMU, we should
perhaps consider changing that.

Regards
Stefan

Re: [Qemu-devel] Qemu 2.2.1 black screen of death in windows 2012 R2

[Stefan Priebe]

Hi,

it started to work again with virtio 100 instead of 94. No idea why it 
works with qemu 2.2.0.


Stefan

Am 24.03.2015 um 12:15 schrieb Stefan Priebe - Profihost AG:


Am 24.03.2015 um 11:45 schrieb Paolo Bonzini:



On 24/03/2015 11:39, Stefan Priebe - Profihost AG wrote:

after upgrading Qemu from 2.2.0 to 2.2.1

Windows 2012 R2 works after installing. But after applying 72 updates it
breaks with a black screen of death.


Can you bisect it?


Have to try might be possible.


Linking to this KB:
https://support.microsoft.com/en-us/kb/2939259


That KB, I think, refers to running a hypervisor (e.g. VMware
Workstation) _inside_ Windows.


Screenshot of the windows screen attached. This is just a blank windows
2012 R2 with virtio 94 drivers installed.

Stefan

Re: [Qemu-devel] Qemu 2.2.1 black screen of death in windows 2012 R2

[Stefan Priebe - Profihost AG]

Am 24.03.2015 um 11:45 schrieb Paolo Bonzini:
 
 
 On 24/03/2015 11:39, Stefan Priebe - Profihost AG wrote:
 after upgrading Qemu from 2.2.0 to 2.2.1

 Windows 2012 R2 works after installing. But after applying 72 updates it
 breaks with a black screen of death.
 
 Can you bisect it?

Have to try might be possible.

 Linking to this KB:
 https://support.microsoft.com/en-us/kb/2939259
 
 That KB, I think, refers to running a hypervisor (e.g. VMware
 Workstation) _inside_ Windows.

Screenshot of the windows screen attached. This is just a blank windows
2012 R2 with virtio 94 drivers installed.

Stefan

[Qemu-devel] Qemu 2.2.1 black screen of death in windows 2012 R2

[Stefan Priebe - Profihost AG]

Hi,

after upgrading Qemu from 2.2.0 to 2.2.1

Windows 2012 R2 works after installing. But after applying 72 updates it
breaks with a black screen of death.

Linking to this KB:
https://support.microsoft.com/en-us/kb/2939259

It works fine with Qemu 2.2.0

Greets,
Stefan

Re: [Qemu-devel] live migration fails after host kernel upgrade (3.12 = 3.18)

[Stefan Priebe - Profihost AG]

Hi,

Thanks.

I fixed it - there is already a patchseries in 4.0 to fix this. It will
be backported in 3.18.10 or 3.18.11.

Stefan

Am 23.03.2015 um 12:54 schrieb Stefan Hajnoczi:
 On Sun, Mar 15, 2015 at 09:59:25AM +0100, Stefan Priebe wrote:
 after upgrading the host kernel from 3.12 to 3.18 live migration fails with
 the following qemu output (guest running on a host with 3.12 = host with
 3.18):

 kvm: Features 0x30afffe3 unsupported. Allowed features: 0x79bfbbe7
 qemu: warning: error while loading state for instance 0x0 of device
 ':00:12.0/virtio-net'
 kvm: load of migration failed: Operation not permitted

 But i can't
 
 Hi Stefan,
 I haven't checked the exact feature bits but it might be the UFO feature
 problem.  The tun driver was changed to drop a feature, this broke live
 migration.
 
 Take a look at
 https://lists.linuxfoundation.org/pipermail/virtualization/2015-February/029217.html
 
 Stefan
 

[Qemu-devel] live migration fails after host kernel upgrade (3.12 = 3.18)

[Stefan Priebe]

Hi,

after upgrading the host kernel from 3.12 to 3.18 live migration fails 
with the following qemu output (guest running on a host with 3.12 = 
host with 3.18):


kvm: Features 0x30afffe3 unsupported. Allowed features: 0x79bfbbe7
qemu: warning: error while loading state for instance 0x0 of device 
':00:12.0/virtio-net'

kvm: load of migration failed: Operation not permitted

But i can't

Greets,
Stefan

Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2

[Stefan Priebe]

Am 16.02.2015 um 16:50 schrieb Andreas Färber:

Am 16.02.2015 um 16:41 schrieb Stefan Priebe - Profihost AG:

Am 16.02.2015 um 15:49 schrieb Paolo Bonzini pbonz...@redhat.com:



On 16/02/2015 15:47, Stefan Priebe - Profihost AG wrote:


Could it be that this is a results of compiling qemu with --enable-debug
to get debugging symbols?


Yes.

*urg* my fault - sorry. Is there a way to get debugging symbols without
using enable-debug / getting slower performance?


Yes, just do nothing (--enable-debug-info is the default;
--enable-debug enables debug info _and_ turns off optimization).


If I do not enable-debug dh_strip does not extract any debugging symbols.


Maybe you forgot --disable-strip?


Thanks!

Stefan



Regards,
Andreas

Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2

[Stefan Priebe]

Am 16.02.2015 um 16:49 schrieb Paolo Bonzini:



On 16/02/2015 16:41, Stefan Priebe - Profihost AG wrote:

Yes, just do nothing (--enable-debug-info is the default;
--enable-debug enables debug info _and_ turns off optimization).

If I do not enable-debug dh_strip does not extract any debugging symbols.


Then you need --disable-strip too.


Thanks!

Stefan



Paolo

Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2

[Stefan Priebe - Profihost AG]

Hi,

Am 16.02.2015 um 13:24 schrieb Paolo Bonzini:
 
 
 On 15/02/2015 19:46, Stefan Priebe wrote:

 while i get a constant random 4k i/o write speed of 20.000 iops with
 qemu 2.1.0 or 2.1.3. I get jumping speeds with qemu 2.2 (jumping between
 500 iops and 15.000 iop/s).

 If i use virtio instead of virtio-scsi speed is the same between 2.2 and
 2.1.
 
 http://wiki.qemu.org/Contribute/ReportABug
 
 What is your command line?
 
 Paolo
 

Could it be that this is a results of compiling qemu with --enable-debug
to get debugging symbols?

Stefan

Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2

[Stefan Priebe - Profihost AG]

Hi,
Am 16.02.2015 um 15:44 schrieb Paolo Bonzini:
 
 
 On 16/02/2015 15:43, Stefan Priebe - Profihost AG wrote:
 Hi,

 Am 16.02.2015 um 13:24 schrieb Paolo Bonzini:


 On 15/02/2015 19:46, Stefan Priebe wrote:

 while i get a constant random 4k i/o write speed of 20.000 iops with
 qemu 2.1.0 or 2.1.3. I get jumping speeds with qemu 2.2 (jumping between
 500 iops and 15.000 iop/s).

 If i use virtio instead of virtio-scsi speed is the same between 2.2 and
 2.1.

 http://wiki.qemu.org/Contribute/ReportABug

 What is your command line?

 Could it be that this is a results of compiling qemu with --enable-debug
 to get debugging symbols?
 
 Yes.

*urg* my fault - sorry. Is there a way to get debugging symbols without
using enable-debug / getting slower performance?

Greets,
Stefan

Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2

[Stefan Priebe - Profihost AG]

Hi,

 Am 16.02.2015 um 15:58 schrieb Andrey Korolyov and...@xdel.ru:
 
 On Mon, Feb 16, 2015 at 5:47 PM, Stefan Priebe - Profihost AG
 s.pri...@profihost.ag wrote:
 Hi,
 Am 16.02.2015 um 15:44 schrieb Paolo Bonzini:
 
 
 On 16/02/2015 15:43, Stefan Priebe - Profihost AG wrote:
 Hi,
 
 Am 16.02.2015 um 13:24 schrieb Paolo Bonzini:
 
 
 On 15/02/2015 19:46, Stefan Priebe wrote:
 
 while i get a constant random 4k i/o write speed of 20.000 iops with
 qemu 2.1.0 or 2.1.3. I get jumping speeds with qemu 2.2 (jumping between
 500 iops and 15.000 iop/s).
 
 If i use virtio instead of virtio-scsi speed is the same between 2.2 and
 2.1.
 
 http://wiki.qemu.org/Contribute/ReportABug
 
 What is your command line?
 
 Could it be that this is a results of compiling qemu with --enable-debug
 to get debugging symbols?
 
 Yes.
 
 *urg* my fault - sorry. Is there a way to get debugging symbols without
 using enable-debug / getting slower performance?
 
 Hi Stefan, splitdebug will help you there. Also this is a standard way
 to ship debugging symbols in most distribuitions, so I wonder if you
 may use just a generic build skeleton for this task...

I'm using enable-debug with dh_strip from Debian

Stefan

Re: [Qemu-devel] slow speed for virtio-scsi since qemu 2.2

[Stefan Priebe - Profihost AG]

 Am 16.02.2015 um 15:49 schrieb Paolo Bonzini pbonz...@redhat.com:
 
 
 
 On 16/02/2015 15:47, Stefan Priebe - Profihost AG wrote:
 
 Could it be that this is a results of compiling qemu with --enable-debug
 to get debugging symbols?
 
 Yes.
 *urg* my fault - sorry. Is there a way to get debugging symbols without
 using enable-debug / getting slower performance?
 
 Yes, just do nothing (--enable-debug-info is the default;
 --enable-debug enables debug info _and_ turns off optimization).

If I do not enable-debug dh_strip does not extract any debugging symbols.

Stefan 

 
 Paolo

[Qemu-devel] slow speed for virtio-scsi since qemu 2.2

[Stefan Priebe]

Hi,

while i get a constant random 4k i/o write speed of 20.000 iops with 
qemu 2.1.0 or 2.1.3. I get jumping speeds with qemu 2.2 (jumping between 
500 iops and 15.000 iop/s).


If i use virtio instead of virtio-scsi speed is the same between 2.2 and 
2.1.


Is there a known regression?

Greets,
Stefan

[Qemu-devel] kvm segfault in iov_to_buf / virtio_scsi_parse_req

[Stefan Priebe]

Hi,

while migrating a bunch of VMs i saw multiple times segaults with qemu 
2.1.2.


Is this a known bug?

Full backtrace:
Program terminated with signal 11, Segmentation fault.
#0  0x7ff9c73bca90 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x7ff9c73bca90 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x7ff9cde46b8e in iov_to_buf (iov=0x7ff9b805c0b0, iov_cnt=1, 
offset=0, buf=0x7ff9b8060118, bytes=51) at util/iov.c:49
#2  0x7ff9cdacdc56 in virtio_scsi_parse_req (req=0x7ff9b8054090, 
req_size=51, resp_size=108)

at qemu-kvm/hw/scsi/virtio-scsi.c:140
#3  0x7ff9cdace060 in virtio_scsi_load_request (f=0x7ff9b803b620, 
sreq=0x7ff9b8089550)

at qemu-kvm/hw/scsi/virtio-scsi.c:215
#4  0x7ff9cdcf5737 in get_scsi_requests (f=0x7ff9b803b620, 
pv=0x7ff9cfe4a440, size=0) at hw/scsi/scsi-bus.c:1875
#5  0x7ff9cdb76e0c in vmstate_load_state (f=0x7ff9b803b620, 
vmsd=0x7ff9ce24cac0, opaque=0x7ff9cfe4a440, version_id=1)

at vmstate.c:105
#6  0x7ff9cdb76de8 in vmstate_load_state (f=0x7ff9b803b620, 
vmsd=0x7ff9ce24c500, opaque=0x7ff9cfe4a440, version_id=1)

at vmstate.c:102
#7  0x7ff9cda935b6 in vmstate_load (f=0x7ff9b803b620, 
se=0x7ff9cfe95570, version_id=1)

at qemu-kvm/savevm.c:572
#8  0x7ff9cda94652 in qemu_loadvm_state (f=0x7ff9b803b620)
at qemu-kvm/savevm.c:951
#9  0x7ff9cdb748bd in process_incoming_migration_co 
(opaque=0x7ff9b803b620) at migration.c:97
#10 0x7ff9cdddb433 in coroutine_trampoline (i0=-1207852976, 
i1=32761) at coroutine-ucontext.c:118

#11 0x7ff9c72dbd10 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x7fff5e55a830 in ?? ()
#13 0x in ?? ()

Stefan

Re: [Qemu-devel] [PATCH] aio: fix qemu_bh_schedule() bh-ctx race condition

[Stefan Priebe - Profihost AG]

Tested-by: Stefan Priebe s.pri...@profihost.ag

Am 03.06.2014 11:21, schrieb Stefan Hajnoczi:
 qemu_bh_schedule() is supposed to be thread-safe at least the first time
 it is called.  Unfortunately this is not quite true:
 
   bh-scheduled = 1;
   aio_notify(bh-ctx);
 
 Since another thread may run the BH callback once it has been scheduled,
 there is a race condition if the callback frees the BH before
 aio_notify(bh-ctx) has a chance to run.
 
 Reported-by: Stefan Priebe s.pri...@profihost.ag
 Signed-off-by: Stefan Hajnoczi stefa...@redhat.com
 ---
  async.c | 14 ++
  1 file changed, 10 insertions(+), 4 deletions(-)
 
 diff --git a/async.c b/async.c
 index 6930185..5b6fe6b 100644
 --- a/async.c
 +++ b/async.c
 @@ -117,15 +117,21 @@ void qemu_bh_schedule_idle(QEMUBH *bh)
  
  void qemu_bh_schedule(QEMUBH *bh)
  {
 +AioContext *ctx;
 +
  if (bh-scheduled)
  return;
 +ctx = bh-ctx;
  bh-idle = 0;
 -/* Make sure that idle  any writes needed by the callback are done
 - * before the locations are read in the aio_bh_poll.
 +/* Make sure that:
 + * 1. idle  any writes needed by the callback are done before the
 + *locations are read in the aio_bh_poll.
 + * 2. ctx is loaded before scheduled is set and the callback has a chance
 + *to execute.
   */
 -smp_wmb();
 +smp_mb();
  bh-scheduled = 1;
 -aio_notify(bh-ctx);
 +aio_notify(ctx);
  }
  
  
 

Re: [Qemu-devel] qemu 2.0 segfaults in event notifier

[Stefan Priebe - Profihost AG]

 Am 02.06.2014 um 15:40 schrieb Stefan Hajnoczi stefa...@gmail.com:
 
 On Fri, May 30, 2014 at 04:10:39PM +0200, Stefan Priebe wrote:
 even with
 +From 271c0f68b4eae72691721243a1c37f46a3232d61 Mon Sep 17 00:00:00 2001
 +From: Fam Zheng f...@redhat.com
 +Date: Wed, 21 May 2014 10:42:13 +0800
 +Subject: [PATCH] aio: Fix use-after-free in cancellation path
 
 applied i saw today segfault with the following backtrace:
 
 Program terminated with signal 11, Segmentation fault.
 #0  0x7f9dd633343f in event_notifier_set (e=0x124) at 
 util/event_notifier-posix.c:97
 97  util/event_notifier-posix.c: No such file or directory.
 (gdb) bt
 #0  0x7f9dd633343f in event_notifier_set (e=0x124) at 
 util/event_notifier-posix.c:97
 #1  0x7f9dd5f4eafc in aio_notify (ctx=0x0) at async.c:246
 #2  0x7f9dd5f4e697 in qemu_bh_schedule (bh=0x7f9b98eeeb30) at async.c:128
 #3  0x7f9dd5fa2c44 in rbd_finish_aiocb (c=0x7f9dd9069ad0, 
 rcb=0x7f9dd85f1770) at block/rbd.c:585
 
 Hi Stefan,
 Please print the QEMUBH:
 (gdb) p *(QEMUBH*)0x7f9b98eeeb30
 
 It would also be interesting to print out the qemu_aio_context-first_bh
 linked list of QEMUBH structs to check whether 0x7f9b98eeeb30 is on the
 list.
 
 The aio_bh_new() and aio_bh_schedule() APIs are supposed to be
 thread-safe.  In theory the rbd.c code is fine.  But maybe there is a
 race condition somewhere.
 
 If you want to debug interactively, ping me on #qemu on irc.oftc.net.

Hi,

that would be great what's your username? On trip right now. Will be on irc in 
4-5 hours or tomorrow in 16 hours.

Greets,
Stefan

Re: [Qemu-devel] qemu 2.0 segfaults in event notifier

[Stefan Priebe]

Am 02.06.2014 15:40, schrieb Stefan Hajnoczi:

On Fri, May 30, 2014 at 04:10:39PM +0200, Stefan Priebe wrote:

even with
+From 271c0f68b4eae72691721243a1c37f46a3232d61 Mon Sep 17 00:00:00 2001
+From: Fam Zheng f...@redhat.com
+Date: Wed, 21 May 2014 10:42:13 +0800
+Subject: [PATCH] aio: Fix use-after-free in cancellation path

applied i saw today segfault with the following backtrace:

Program terminated with signal 11, Segmentation fault.
#0  0x7f9dd633343f in event_notifier_set (e=0x124) at 
util/event_notifier-posix.c:97
97  util/event_notifier-posix.c: No such file or directory.
(gdb) bt
#0  0x7f9dd633343f in event_notifier_set (e=0x124) at 
util/event_notifier-posix.c:97
#1  0x7f9dd5f4eafc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f9dd5f4e697 in qemu_bh_schedule (bh=0x7f9b98eeeb30) at async.c:128
#3  0x7f9dd5fa2c44 in rbd_finish_aiocb (c=0x7f9dd9069ad0, 
rcb=0x7f9dd85f1770) at block/rbd.c:585


Hi Stefan,
Please print the QEMUBH:
(gdb) p *(QEMUBH*)0x7f9b98eeeb30


new trace:
(gdb) bt
#0  0x7f69e421c43f in event_notifier_set (e=0x124) at 
util/event_notifier-posix.c:97

#1  0x7f69e3e37afc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f69e3e37697 in qemu_bh_schedule (bh=0x7f5dac217f60) at 
async.c:128
#3  0x7f69e3e8bc44 in rbd_finish_aiocb (c=0x7f5dac0c3f30, 
rcb=0x7f5dafa50610) at block/rbd.c:585
#4  0x7f69e17bee44 in librbd::AioCompletion::complete() () from 
/usr/lib/librbd.so.1
#5  0x7f69e17be832 in 
librbd::AioCompletion::complete_request(CephContext*, long) () from 
/usr/lib/librbd.so.1
#6  0x7f69e1c946ba in Context::complete(int) () from 
/usr/lib/librados.so.2
#7  0x7f69e17f1e85 in ObjectCacher::C_WaitForWrite::finish(int) () 
from /usr/lib/librbd.so.1
#8  0x7f69e1c946ba in Context::complete(int) () from 
/usr/lib/librados.so.2
#9  0x7f69e1d373c8 in Finisher::finisher_thread_entry() () from 
/usr/lib/librados.so.2
#10 0x7f69dbd43b50 in start_thread () from 
/lib/x86_64-linux-gnu/libpthread.so.0

#11 0x7f69dba8e13d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x in ?? ()

this i another core dump so address differ:
(gdb) p *(QEMUBH*)0x7f5dac217f60
$1 = {ctx = 0x0, cb = 0x7f69e3e8bb75 rbd_finish_bh, opaque = 
0x7f5dafa50610, next = 0x7f69e6b04d10, scheduled = false,

  idle = false, deleted = true}


It would also be interesting to print out the qemu_aio_context-first_bh
linked list of QEMUBH structs to check whether 0x7f9b98eeeb30 is on the
list.


Do you mean just this:
(gdb) p *(QEMUBH*)qemu_aio_context-first_bh
$3 = {ctx = 0x7f69e68a4e00, cb = 0x7f69e41546a5 virtio_net_tx_bh, 
opaque = 0x7f69e6b4a5e0, next = 0x7f69e6b4a570,

  scheduled = false, idle = false, deleted = false}


The aio_bh_new() and aio_bh_schedule() APIs are supposed to be
thread-safe.  In theory the rbd.c code is fine.  But maybe there is a
race condition somewhere.


rbd.c was fine with 1.7.0

Stefan

Re: [Qemu-devel] qemu 2.0 segfaults in event notifier

[Stefan Priebe]

Am 02.06.2014 22:45, schrieb Paolo Bonzini:

Il 02/06/2014 21:32, Stefan Priebe ha scritto:


#0  0x7f69e421c43f in event_notifier_set (e=0x124) at
util/event_notifier-posix.c:97
#1  0x7f69e3e37afc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f69e3e37697 in qemu_bh_schedule (bh=0x7f5dac217f60) at
async.c:128
#3  0x7f69e3e8bc44 in rbd_finish_aiocb (c=0x7f5dac0c3f30,
rcb=0x7f5dafa50610) at block/rbd.c:585
#4  0x7f69e17bee44 in librbd::AioCompletion::complete() () from
/usr/lib/librbd.so.1
#5  0x7f69e17be832 in
librbd::AioCompletion::complete_request(CephContext*, long) () from
/usr/lib/librbd.so.1
#6  0x7f69e1c946ba in Context::complete(int) () from
/usr/lib/librados.so.2
#7  0x7f69e17f1e85 in ObjectCacher::C_WaitForWrite::finish(int) ()
from /usr/lib/librbd.so.1
#8  0x7f69e1c946ba in Context::complete(int) () from
/usr/lib/librados.so.2
#9  0x7f69e1d373c8 in Finisher::finisher_thread_entry() () from
/usr/lib/librados.so.2
#10 0x7f69dbd43b50 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x7f69dba8e13d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x in ?? ()


Can you also print qemu_aio_context?


(gdb) print  qemu_aio_context
$1 = (AioContext *) 0x7f69e68a4e00

(gdb) print *(AioContext*)0x7f69e68a4e00
$2 = {source = {callback_data = 0x0, callback_funcs = 0x0, source_funcs 
= 0x7f69e462d020, ref_count = 2, context = 0x7f69e68a5190,
priority = 0, flags = 1, source_id = 1, poll_fds = 0x7f69e686aea0, 
prev = 0x0, next = 0x7f69e743ccd0, name = 0x0, priv = 0x0},
  lock = {lock = {lock = {__data = {__lock = 0, __count = 0, __owner = 
0, __nusers = 0, __kind = 2, __spins = 0, __list = {
__prev = 0x0, __next = 0x0}}, __size = '\000' repeats 16 
times, \002, '\000' repeats 22 times, __align = 0}},
head = 0, tail = 0, cond = {cond = {__data = {__lock = 0, __futex = 
0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0,
  __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = 
'\000' repeats 47 times, __align = 0}}, owner_thread = {
  thread = 0}, nesting = 0, cb = 0x7f69e3e37b4f aio_rfifolock_cb, 
cb_opaque = 0x7f69e68a4e00}, aio_handlers = {
lh_first = 0x7f69e68a4f60}, walking_handlers = 0, bh_lock = {lock = 
{__data = {__lock = 0, __count = 0, __owner = 0,
__nusers = 0, __kind = 2, __spins = 0, __list = {__prev = 0x0, 
__next = 0x0}},
  __size = '\000' repeats 16 times, \002, '\000' repeats 22 
times, __align = 0}}, first_bh = 0x7f69e6b04d10,
  walking_bh = 0, notifier = {rfd = 4, wfd = 4}, pollfds = 
0x7f69e68a4630, thread_pool = 0x0, tlg = {tl = {0x7f69e68a4fa0,

  0x7f69e68a5010, 0x7f69e68a5080}}}


 Also print the backtrace of all
threads, using thread apply all bt full.


http://pastebin.com/raw.php?i=uzcpN0zk

THanks,
Stefan

Re: [Qemu-devel] [Qemu-stable] qemu segfault in librbd / aio_notify

[Stefan Priebe]

Hi,

even with
+From 271c0f68b4eae72691721243a1c37f46a3232d61 Mon Sep 17 00:00:00 2001
+From: Fam Zheng f...@redhat.com
+Date: Wed, 21 May 2014 10:42:13 +0800
+Subject: [PATCH] aio: Fix use-after-free in cancellation path

applied i saw today segfault with the following backtrace:

Program terminated with signal 11, Segmentation fault.
#0  0x7f9dd633343f in event_notifier_set (e=0x124) at 
util/event_notifier-posix.c:97

97  util/event_notifier-posix.c: No such file or directory.
(gdb) bt
#0  0x7f9dd633343f in event_notifier_set (e=0x124) at 
util/event_notifier-posix.c:97

#1  0x7f9dd5f4eafc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f9dd5f4e697 in qemu_bh_schedule (bh=0x7f9b98eeeb30) at 
async.c:128
#3  0x7f9dd5fa2c44 in rbd_finish_aiocb (c=0x7f9dd9069ad0, 
rcb=0x7f9dd85f1770) at block/rbd.c:585
#4  0x7f9dd38d5e44 in librbd::AioCompletion::complete() () from 
/usr/lib/librbd.so.1
#5  0x7f9dd38d5832 in 
librbd::AioCompletion::complete_request(CephContext*, long) () from 
/usr/lib/librbd.so.1
#6  0x7f9dd3dab6ba in Context::complete(int) () from 
/usr/lib/librados.so.2
#7  0x7f9dd3908e85 in ObjectCacher::C_WaitForWrite::finish(int) () 
from /usr/lib/librbd.so.1
#8  0x7f9dd3dab6ba in Context::complete(int) () from 
/usr/lib/librados.so.2
#9  0x7f9dd3e4e3c8 in Finisher::finisher_thread_entry() () from 
/usr/lib/librados.so.2
#10 0x7f9dcde5ab50 in start_thread () from 
/lib/x86_64-linux-gnu/libpthread.so.0

#11 0x7f9dcdba513d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x in ?? ()


Am 28.05.2014 21:44, schrieb Stefan Priebe:

is this:
commit 271c0f68b4eae72691721243a1c37f46a3232d61
Author: Fam Zheng f...@redhat.com
Date:   Wed May 21 10:42:13 2014 +0800

 aio: Fix use-after-free in cancellation path

Stefan

Am 28.05.2014 21:40, schrieb Stefan Priebe:

Hello,

i mean since using qemu 2.0 i've now seen several times the following
segfault:
(gdb) bt
#0  0x7f2af1196433 in event_notifier_set (e=0x124) at
util/event_notifier-posix.c:97
#1  0x7f2af0db1afc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f2af0db1697 in qemu_bh_schedule (bh=0x7f2ad401bec0) at
async.c:128
#3  0x7f2af0e05c44 in rbd_finish_aiocb (c=0x7f2ad5ec4590,
rcb=0x7f2ad63c5df0) at block/rbd.c:585
#4  0x7f2aee738e44 in librbd::AioCompletion::complete() () from
/usr/lib/librbd.so.1
#5  0x7f2aee738832 in
librbd::AioCompletion::complete_request(CephContext*, long) () from
/usr/lib/librbd.so.1
#6  0x7f2aeec0e6ba in Context::complete(int) () from
/usr/lib/librados.so.2
#7  0x7f2aee76be85 in ObjectCacher::C_WaitForWrite::finish(int) ()
from /usr/lib/librbd.so.1
#8  0x7f2aeec0e6ba in Context::complete(int) () from
/usr/lib/librados.so.2
#9  0x7f2aeecb13c8 in Finisher::finisher_thread_entry() () from
/usr/lib/librados.so.2
#10 0x7f2ae8cbdb50 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x7f2ae8a080ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x in ?? ()
(gdb)


from another VM:
#0  0x7f89565ec433 in event_notifier_set (e=0x124) at
util/event_notifier-posix.c:97
#1  0x7f8956207afc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f8956207697 in qemu_bh_schedule (bh=0x7f882dd6d340) at
async.c:128
#3  0x7f895625bc44 in rbd_finish_aiocb (c=0x7f882d4c34a0,
rcb=0x7f882c0ae350) at block/rbd.c:585
#4  0x7f8953b8ee44 in librbd::AioCompletion::complete() () from
/usr/lib/librbd.so.1
#5  0x7f8953b8e832 in
librbd::AioCompletion::complete_request(CephContext*, long) () from
/usr/lib/librbd.so.1
#6  0x7f89540646ba in Context::complete(int) () from
/usr/lib/librados.so.2
#7  0x7f8953bc1e85 in ObjectCacher::C_WaitForWrite::finish(int) ()
from /usr/lib/librbd.so.1
#8  0x7f89540646ba in Context::complete(int) () from
/usr/lib/librados.so.2
#9  0x7f89541073c8 in Finisher::finisher_thread_entry() () from
/usr/lib/librados.so.2
#10 0x7f894e113b50 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x7f894de5e0ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x in ?? ()

Stefan

[Qemu-devel] qemu 2.0 segfaults in event notifier

[Stefan Priebe]

Hi,

even with
+From 271c0f68b4eae72691721243a1c37f46a3232d61 Mon Sep 17 00:00:00 2001
+From: Fam Zheng f...@redhat.com
+Date: Wed, 21 May 2014 10:42:13 +0800
+Subject: [PATCH] aio: Fix use-after-free in cancellation path

applied i saw today segfault with the following backtrace:

Program terminated with signal 11, Segmentation fault.
#0  0x7f9dd633343f in event_notifier_set (e=0x124) at 
util/event_notifier-posix.c:97
97  util/event_notifier-posix.c: No such file or directory.
(gdb) bt
#0  0x7f9dd633343f in event_notifier_set (e=0x124) at 
util/event_notifier-posix.c:97
#1  0x7f9dd5f4eafc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f9dd5f4e697 in qemu_bh_schedule (bh=0x7f9b98eeeb30) at async.c:128
#3  0x7f9dd5fa2c44 in rbd_finish_aiocb (c=0x7f9dd9069ad0, 
rcb=0x7f9dd85f1770) at block/rbd.c:585
#4  0x7f9dd38d5e44 in librbd::AioCompletion::complete() () from 
/usr/lib/librbd.so.1
#5  0x7f9dd38d5832 in librbd::AioCompletion::complete_request(CephContext*, 
long) () from /usr/lib/librbd.so.1
#6  0x7f9dd3dab6ba in Context::complete(int) () from /usr/lib/librados.so.2
#7  0x7f9dd3908e85 in ObjectCacher::C_WaitForWrite::finish(int) () from 
/usr/lib/librbd.so.1
#8  0x7f9dd3dab6ba in Context::complete(int) () from /usr/lib/librados.so.2
#9  0x7f9dd3e4e3c8 in Finisher::finisher_thread_entry() () from 
/usr/lib/librados.so.2
#10 0x7f9dcde5ab50 in start_thread () from 
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x7f9dcdba513d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x in ?? ()


Am 28.05.2014 21:44, schrieb Stefan Priebe:
 is this:
 commit 271c0f68b4eae72691721243a1c37f46a3232d61
 Author: Fam Zheng f...@redhat.com
 Date:   Wed May 21 10:42:13 2014 +0800
 
 aio: Fix use-after-free in cancellation path
 
 Stefan
 
 Am 28.05.2014 21:40, schrieb Stefan Priebe:
 Hello,
 
 i mean since using qemu 2.0 i've now seen several times the following
 segfault:
 (gdb) bt
 #0  0x7f2af1196433 in event_notifier_set (e=0x124) at
 util/event_notifier-posix.c:97
 #1  0x7f2af0db1afc in aio_notify (ctx=0x0) at async.c:246
 #2  0x7f2af0db1697 in qemu_bh_schedule (bh=0x7f2ad401bec0) at
 async.c:128
 #3  0x7f2af0e05c44 in rbd_finish_aiocb (c=0x7f2ad5ec4590,
 rcb=0x7f2ad63c5df0) at block/rbd.c:585
 #4  0x7f2aee738e44 in librbd::AioCompletion::complete() () from
 /usr/lib/librbd.so.1
 #5  0x7f2aee738832 in
 librbd::AioCompletion::complete_request(CephContext*, long) () from
 /usr/lib/librbd.so.1
 #6  0x7f2aeec0e6ba in Context::complete(int) () from
 /usr/lib/librados.so.2
 #7  0x7f2aee76be85 in ObjectCacher::C_WaitForWrite::finish(int) ()
 from /usr/lib/librbd.so.1
 #8  0x7f2aeec0e6ba in Context::complete(int) () from
 /usr/lib/librados.so.2
 #9  0x7f2aeecb13c8 in Finisher::finisher_thread_entry() () from
 /usr/lib/librados.so.2
 #10 0x7f2ae8cbdb50 in start_thread () from
 /lib/x86_64-linux-gnu/libpthread.so.0
 #11 0x7f2ae8a080ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
 #12 0x in ?? ()
 (gdb)
 
 
 from another VM:
 #0  0x7f89565ec433 in event_notifier_set (e=0x124) at
 util/event_notifier-posix.c:97
 #1  0x7f8956207afc in aio_notify (ctx=0x0) at async.c:246
 #2  0x7f8956207697 in qemu_bh_schedule (bh=0x7f882dd6d340) at
 async.c:128
 #3  0x7f895625bc44 in rbd_finish_aiocb (c=0x7f882d4c34a0,
 rcb=0x7f882c0ae350) at block/rbd.c:585
 #4  0x7f8953b8ee44 in librbd::AioCompletion::complete() () from
 /usr/lib/librbd.so.1
 #5  0x7f8953b8e832 in
 librbd::AioCompletion::complete_request(CephContext*, long) () from
 /usr/lib/librbd.so.1
 #6  0x7f89540646ba in Context::complete(int) () from
 /usr/lib/librados.so.2
 #7  0x7f8953bc1e85 in ObjectCacher::C_WaitForWrite::finish(int) ()
 from /usr/lib/librbd.so.1
 #8  0x7f89540646ba in Context::complete(int) () from
 /usr/lib/librados.so.2
 #9  0x7f89541073c8 in Finisher::finisher_thread_entry() () from
 /usr/lib/librados.so.2
 #10 0x7f894e113b50 in start_thread () from
 /lib/x86_64-linux-gnu/libpthread.so.0
 #11 0x7f894de5e0ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
 #12 0x in ?? ()
 
 Stefan
 

[Qemu-devel] qemu segfault in librbd / aio_notify

[Stefan Priebe]

Hello,

i mean since using qemu 2.0 i've now seen several times the following 
segfault:

(gdb) bt
#0  0x7f2af1196433 in event_notifier_set (e=0x124) at 
util/event_notifier-posix.c:97

#1  0x7f2af0db1afc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f2af0db1697 in qemu_bh_schedule (bh=0x7f2ad401bec0) at 
async.c:128
#3  0x7f2af0e05c44 in rbd_finish_aiocb (c=0x7f2ad5ec4590, 
rcb=0x7f2ad63c5df0) at block/rbd.c:585
#4  0x7f2aee738e44 in librbd::AioCompletion::complete() () from 
/usr/lib/librbd.so.1
#5  0x7f2aee738832 in 
librbd::AioCompletion::complete_request(CephContext*, long) () from 
/usr/lib/librbd.so.1
#6  0x7f2aeec0e6ba in Context::complete(int) () from 
/usr/lib/librados.so.2
#7  0x7f2aee76be85 in ObjectCacher::C_WaitForWrite::finish(int) () 
from /usr/lib/librbd.so.1
#8  0x7f2aeec0e6ba in Context::complete(int) () from 
/usr/lib/librados.so.2
#9  0x7f2aeecb13c8 in Finisher::finisher_thread_entry() () from 
/usr/lib/librados.so.2
#10 0x7f2ae8cbdb50 in start_thread () from 
/lib/x86_64-linux-gnu/libpthread.so.0

#11 0x7f2ae8a080ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x in ?? ()
(gdb)


from another VM:
#0  0x7f89565ec433 in event_notifier_set (e=0x124) at 
util/event_notifier-posix.c:97

#1  0x7f8956207afc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f8956207697 in qemu_bh_schedule (bh=0x7f882dd6d340) at 
async.c:128
#3  0x7f895625bc44 in rbd_finish_aiocb (c=0x7f882d4c34a0, 
rcb=0x7f882c0ae350) at block/rbd.c:585
#4  0x7f8953b8ee44 in librbd::AioCompletion::complete() () from 
/usr/lib/librbd.so.1
#5  0x7f8953b8e832 in 
librbd::AioCompletion::complete_request(CephContext*, long) () from 
/usr/lib/librbd.so.1
#6  0x7f89540646ba in Context::complete(int) () from 
/usr/lib/librados.so.2
#7  0x7f8953bc1e85 in ObjectCacher::C_WaitForWrite::finish(int) () 
from /usr/lib/librbd.so.1
#8  0x7f89540646ba in Context::complete(int) () from 
/usr/lib/librados.so.2
#9  0x7f89541073c8 in Finisher::finisher_thread_entry() () from 
/usr/lib/librados.so.2
#10 0x7f894e113b50 in start_thread () from 
/lib/x86_64-linux-gnu/libpthread.so.0

#11 0x7f894de5e0ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x in ?? ()

Stefan

Re: [Qemu-devel] [Qemu-stable] qemu segfault in librbd / aio_notify

[Stefan Priebe]

is this:
commit 271c0f68b4eae72691721243a1c37f46a3232d61
Author: Fam Zheng f...@redhat.com
Date:   Wed May 21 10:42:13 2014 +0800

aio: Fix use-after-free in cancellation path

Stefan

Am 28.05.2014 21:40, schrieb Stefan Priebe:

Hello,

i mean since using qemu 2.0 i've now seen several times the following
segfault:
(gdb) bt
#0  0x7f2af1196433 in event_notifier_set (e=0x124) at
util/event_notifier-posix.c:97
#1  0x7f2af0db1afc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f2af0db1697 in qemu_bh_schedule (bh=0x7f2ad401bec0) at
async.c:128
#3  0x7f2af0e05c44 in rbd_finish_aiocb (c=0x7f2ad5ec4590,
rcb=0x7f2ad63c5df0) at block/rbd.c:585
#4  0x7f2aee738e44 in librbd::AioCompletion::complete() () from
/usr/lib/librbd.so.1
#5  0x7f2aee738832 in
librbd::AioCompletion::complete_request(CephContext*, long) () from
/usr/lib/librbd.so.1
#6  0x7f2aeec0e6ba in Context::complete(int) () from
/usr/lib/librados.so.2
#7  0x7f2aee76be85 in ObjectCacher::C_WaitForWrite::finish(int) ()
from /usr/lib/librbd.so.1
#8  0x7f2aeec0e6ba in Context::complete(int) () from
/usr/lib/librados.so.2
#9  0x7f2aeecb13c8 in Finisher::finisher_thread_entry() () from
/usr/lib/librados.so.2
#10 0x7f2ae8cbdb50 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x7f2ae8a080ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x in ?? ()
(gdb)


from another VM:
#0  0x7f89565ec433 in event_notifier_set (e=0x124) at
util/event_notifier-posix.c:97
#1  0x7f8956207afc in aio_notify (ctx=0x0) at async.c:246
#2  0x7f8956207697 in qemu_bh_schedule (bh=0x7f882dd6d340) at
async.c:128
#3  0x7f895625bc44 in rbd_finish_aiocb (c=0x7f882d4c34a0,
rcb=0x7f882c0ae350) at block/rbd.c:585
#4  0x7f8953b8ee44 in librbd::AioCompletion::complete() () from
/usr/lib/librbd.so.1
#5  0x7f8953b8e832 in
librbd::AioCompletion::complete_request(CephContext*, long) () from
/usr/lib/librbd.so.1
#6  0x7f89540646ba in Context::complete(int) () from
/usr/lib/librados.so.2
#7  0x7f8953bc1e85 in ObjectCacher::C_WaitForWrite::finish(int) ()
from /usr/lib/librbd.so.1
#8  0x7f89540646ba in Context::complete(int) () from
/usr/lib/librados.so.2
#9  0x7f89541073c8 in Finisher::finisher_thread_entry() () from
/usr/lib/librados.so.2
#10 0x7f894e113b50 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#11 0x7f894de5e0ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x in ?? ()

Stefan

Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work

[Stefan Priebe - Profihost AG]

Hi,

i now was able to catch the error.

It is:
Length mismatch: :00:12.0/virtio-net-pci.rom: 4 in != 1
qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed

Stefan

Am 09.05.2014 19:05, schrieb Paolo Bonzini:
 Il 09/05/2014 15:13, Stefan Priebe - Profihost AG ha scritto:

 I see no output at the monitor of Qemu 2.0.

 # migrate -d tcp:a.b.c.d:6000

 # info migrate
 capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on
 zero-blocks: off
 Migration status: failed
 total time: 0 milliseconds

 The target machine is still running at this point with no output.
 
 Anything on its stdout?
 
 Another test you could do, in addition to changing the devices, is this:
 
 1) try with commit 6141f3bd6904df7cf9519c6444a14a608b9874c4 on the
 destination (the next one caused a migration problem that was fixed
 later).  If it passes, go to step 1a.  If it fails, go to step 2.
 
 1a) try with commit c01a71c1a56fa27f43449ff59e5d03b2483658a2.  If it
 passes, go to step 1b.  If it fails, report it here.
 
 1b) bisect between v2.0.0 (known-bad) and
 c01a71c1a56fa27f43449ff59e5d03b2483658a2 (known-good) to find the
 culprit.  Report results.
 
 2) change the source to v1.7.0 and bisect between v1.7.0 and
 6141f3bd6904df7cf9519c6444a14a608b9874c4.  Report results.
 
 Thanks!
 
 Paolo

Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work

[Stefan Priebe - Profihost AG]

Am 14.05.2014 10:11, schrieb Paolo Bonzini:
 Il 14/05/2014 09:17, Stefan Priebe - Profihost AG ha scritto:
 i now was able to catch the error.

 It is:
 Length mismatch: :00:12.0/virtio-net-pci.rom: 4 in != 1
 qemu: warning: error while loading state for instance 0x0 of device 'ram'
 load of migration failed
 
 This is a bug of your distribution.  The file sizes for ROMs should
 never change.  In particular, if you round the sizes up to the next
 power of 2 you should always have:
 
 128k for bios.bin
 256k for bios-256k.bin
 64k for pxe-*.rom
 256k for efi-*.rom
 64k for vgabios-*
 
 Unfortunately, most distribution get pxe-*.rom sizes wrong, because at
 some point iPXE grew more features and didn't fit in 64k anymore with
 the default configuration.  I know at least Fedora does.
 
 The solution is to copy the binaries from the QEMU git repository
 (directory pc-bios/) to /usr/share/qemu.

Hi,

i compile qemu on my own.

I have the rom files under /usr/share/kvm and they look like this:
ls -la /usr/share/kvm/*.rom
-rw-r--r-- 1 root root 173568 May 14 09:39 /usr/share/kvm/efi-e1000.rom
-rw-r--r-- 1 root root 174592 May 14 09:39 /usr/share/kvm/efi-eepro100.rom
-rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-ne2k_pci.rom
-rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-pcnet.rom
-rw-r--r-- 1 root root 176640 May 14 09:39 /usr/share/kvm/efi-rtl8139.rom
-rw-r--r-- 1 root root 171008 May 14 09:39 /usr/share/kvm/efi-virtio.rom
-rw-r--r-- 1 root root  67072 May 14 09:39 /usr/share/kvm/pxe-e1000.rom
-rw-r--r-- 1 root root  61440 May 14 09:39 /usr/share/kvm/pxe-eepro100.rom
-rw-r--r-- 1 root root  61440 May 14 09:39 /usr/share/kvm/pxe-ne2k_pci.rom
-rw-r--r-- 1 root root  61440 May 14 09:39 /usr/share/kvm/pxe-pcnet.rom
-rw-r--r-- 1 root root  61440 May 14 09:39 /usr/share/kvm/pxe-rtl8139.rom
-rw-r--r-- 1 root root  60416 May 14 09:39 /usr/share/kvm/pxe-virtio.rom

currently i don't know what's wrong.

Stefan

Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work

[Stefan Priebe - Profihost AG]

Am 14.05.2014 10:36, schrieb Paolo Bonzini:
 Il 14/05/2014 10:29, Stefan Priebe - Profihost AG ha scritto:
 Hi,

 i compile qemu on my own.

 I have the rom files under /usr/share/kvm and they look like this:
 ls -la /usr/share/kvm/*.rom
 -rw-r--r-- 1 root root 173568 May 14 09:39 /usr/share/kvm/efi-e1000.rom
 -rw-r--r-- 1 root root 174592 May 14 09:39
 /usr/share/kvm/efi-eepro100.rom
 -rw-r--r-- 1 root root 173056 May 14 09:39
 /usr/share/kvm/efi-ne2k_pci.rom
 -rw-r--r-- 1 root root 173056 May 14 09:39 /usr/share/kvm/efi-pcnet.rom
 -rw-r--r-- 1 root root 176640 May 14 09:39 /usr/share/kvm/efi-rtl8139.rom
 -rw-r--r-- 1 root root 171008 May 14 09:39 /usr/share/kvm/efi-virtio.rom
 -rw-r--r-- 1 root root  67072 May 14 09:39 /usr/share/kvm/pxe-e1000.rom
 -rw-r--r-- 1 root root  61440 May 14 09:39
 /usr/share/kvm/pxe-eepro100.rom
 -rw-r--r-- 1 root root  61440 May 14 09:39
 /usr/share/kvm/pxe-ne2k_pci.rom
 -rw-r--r-- 1 root root  61440 May 14 09:39 /usr/share/kvm/pxe-pcnet.rom
 -rw-r--r-- 1 root root  61440 May 14 09:39 /usr/share/kvm/pxe-rtl8139.rom
 -rw-r--r-- 1 root root  60416 May 14 09:39 /usr/share/kvm/pxe-virtio.rom

 currently i don't know what's wrong.
 
 What about the source machine?

Currently it has the same as i already updated the package there too.

So you mean i had done a mistake compiling the old package - so it had
wrong sizes?

Greets.
Stefan

Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work

[Stefan Priebe - Profihost AG]

Am 14.05.2014 11:00, schrieb Paolo Bonzini:
 Il 14/05/2014 10:38, Stefan Priebe - Profihost AG ha scritto:
 Currently it has the same as i already updated the package there too.

 So you mean i had done a mistake compiling the old package - so it had
 wrong sizes?
 
 Yes, probably.
 
 Can you do an info mtree for a machine that's running on the source?
 
 Paolo

here it is:
# info mtree
memory
-7ffe (prio 0, RW): system
  -dfff (prio 0, RW): alias ram-below-4g
@pc.ram -dfff
  000a-000b (prio 1, RW): alias smram-region
@pci 000a-000b
  000c-000c3fff (prio 1, R-): alias pam-rom @pc.ram
000c-000c3fff
  000c4000-000c7fff (prio 1, R-): alias pam-rom @pc.ram
000c4000-000c7fff
  000c8000-000cbfff (prio 1, R-): alias pam-rom @pc.ram
000c8000-000cbfff
  000ca000-000ccfff (prio 1000, RW): alias kvmvapic-rom
@pc.ram 000ca000-000ccfff
  000cc000-000c (prio 1, R-): alias pam-rom @pc.ram
000cc000-000c
  000d-000d3fff (prio 1, RW): alias pam-ram @pc.ram
000d-000d3fff
  000d4000-000d7fff (prio 1, RW): alias pam-ram @pc.ram
000d4000-000d7fff
  000d8000-000dbfff (prio 1, RW): alias pam-ram @pc.ram
000d8000-000dbfff
  000dc000-000d (prio 1, RW): alias pam-ram @pc.ram
000dc000-000d
  000e-000e3fff (prio 1, RW): alias pam-ram @pc.ram
000e-000e3fff
  000e4000-000e7fff (prio 1, RW): alias pam-ram @pc.ram
000e4000-000e7fff
  000e8000-000ebfff (prio 1, RW): alias pam-ram @pc.ram
000e8000-000ebfff
  000ec000-000e (prio 1, RW): alias pam-ram @pc.ram
000ec000-000e
  000f-000f (prio 1, R-): alias pam-rom @pc.ram
000f-000f
  e000- (prio 0, RW): alias pci-hole @pci
e000-
  fec0-fec00fff (prio 0, RW): kvm-ioapic
  fed0-fed003ff (prio 0, RW): hpet
  fee0-feef (prio 4096, RW): icc-apic-container
fee0-feef (prio 0, RW): kvm-apic-msi
  0001-00021fff (prio 0, RW): alias ram-above-4g
@pc.ram e000-0001
  00022000-40021fff (prio 0, RW): alias pci-hole64 @pci
00022000-40021fff
I/O
- (prio 0, RW): io
  -0007 (prio 0, RW): dma-chan
  0008-000f (prio 0, RW): dma-cont
  0020-0021 (prio 0, RW): kvm-pic
  0040-0043 (prio 0, RW): kvm-pit
  0060-0060 (prio 0, RW): i8042-data
  0061-0061 (prio 0, RW): elcr
  0064-0064 (prio 0, RW): i8042-cmd
  0070-0071 (prio 0, RW): rtc
  007e-007f (prio 0, RW): kvmvapic
  0080-0080 (prio 0, RW): ioport80
  0081-0083 (prio 0, RW): alias dma-page
@dma-page 0081-0083
  0087-0087 (prio 0, RW): alias dma-page
@dma-page 0087-0087
  0089-008b (prio 0, RW): alias dma-page
@dma-page 0089-008b
  008f-008f (prio 0, RW): alias dma-page
@dma-page 008f-008f
  0092-0092 (prio 0, RW): port92
  00a0-00a1 (prio 0, RW): kvm-pic
  00b2-00b3 (prio 0, RW): apm-io
  00c0-00cf (prio 0, RW): dma-chan
  00d0-00df (prio 0, RW): dma-cont
  00f0-00f0 (prio 0, RW): ioportF0
  0170-0177 (prio 0, RW): alias ide @ide
0170-0177
  01f0-01f7 (prio 0, RW): alias ide @ide
01f0-01f7
  0376-0376 (prio 0, RW): alias ide @ide
0376-0376
  03b0-03df (prio 0, RW): cirrus-io
  03f1-03f5 (prio 0, RW): alias fdc @fdc
03f1-03f5
  03f6-03f6 (prio 0, RW): alias ide @ide
03f6-03f6
  03f7-03f7 (prio 0, RW): alias fdc @fdc
03f7-03f7
  04d0-04d0 (prio 0, RW): kvm-elcr
  04d1-04d1 (prio 0, RW): kvm-elcr
  0505-0505 (prio 0, RW): pvpanic

[Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work

[Stefan Priebe - Profihost AG]

Hello list,

i was trying to migrate older Qemu (1.5 and 1.7.2) to a machine running
Qemu 2.0.

I started the target machine with:

-machine type=pc-i440fx-1.5 / -machine type=pc-i440fx-1.7

But the migration simply fails. Migrating Qemu 2.0 to Qemu 2.0 succeeds.

I see no output at the monitor of Qemu 2.0.

# migrate -d tcp:a.b.c.d:6000

# info migrate
capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on
zero-blocks: off
Migration status: failed
total time: 0 milliseconds

The target machine is still running at this point with no output.

Stefan

Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work

[Stefan Priebe - Profihost AG]

 Am 09.05.2014 um 15:41 schrieb Dr. David Alan Gilbert dgilb...@redhat.com:
 
 * Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote:
 Hello list,
 
 i was trying to migrate older Qemu (1.5 and 1.7.2) to a machine running
 Qemu 2.0.
 
 I started the target machine with:
 
 -machine type=pc-i440fx-1.5 / -machine type=pc-i440fx-1.7
 
 I'd expect you to have to run with the same machine type on both sides.
 I ran some simple virt-test migrate tests (just the basic one) and got
 v1.5.3-v1.6.2
 v1.5.3-v1.7.1
 v1.5.3-v2.0.0-rc1
   working for most machine types, pc-i440fx-1.5 passed with all of those.
 Note that's only the simplest test.

My test involved USB Controller virtio Network cards and rbd virtio-scsi drives.

Can you give me your simple start line so I could test if this works for me 
while adding some more arguments.

Thanks!

Stefan

 
 Dave
 --
 Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work

[Stefan Priebe]

Am 09.05.2014 18:29, schrieb Dr. David Alan Gilbert:

* Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote:



Am 09.05.2014 um 15:41 schrieb Dr. David Alan Gilbert dgilb...@redhat.com:

* Stefan Priebe - Profihost AG (s.pri...@profihost.ag) wrote:

Hello list,

i was trying to migrate older Qemu (1.5 and 1.7.2) to a machine running
Qemu 2.0.

I started the target machine with:

-machine type=pc-i440fx-1.5 / -machine type=pc-i440fx-1.7


I'd expect you to have to run with the same machine type on both sides.
I ran some simple virt-test migrate tests (just the basic one) and got
v1.5.3-v1.6.2
v1.5.3-v1.7.1
v1.5.3-v2.0.0-rc1
   working for most machine types, pc-i440fx-1.5 passed with all of those.
Note that's only the simplest test.


My test involved USB Controller virtio Network cards and rbd virtio-scsi drives.


That probably makes things more interesting :-) I'd start with the simplest
config you can find and add each one of those until it breaks.


Can you give me your simple start line so I could test if this works for me 
while adding some more arguments.


I've got a slightly hacked up libvirt, but I'm doing

   ./run -t qemu --machine-type=$MACTYPE -g Linux.JeOS.19.x86_64.i440fx 
--qemu-bin=$SRC --qemu-dst-bin=$DST --qemu_sandbox=off --tests 
migrate.default.tcp --disk-bus=ide

and looking at the logs we have:

/opt/qemu/v1.5.3/bin/qemu-system-x86_64 \
 -S  \
 -name 'virt-tests-vm1'  \
 -sandbox off  \
 -M pc-i440fx-1.5  \
 -nodefaults  \
 -vga std  \
 -chardev 
socket,id=hmp_id_hmp1,path=/tmp/monitor-hmp1-20140415-104517-rAfzfDef,server,nowait
 \
 -mon chardev=hmp_id_hmp1,mode=readline  \
 -chardev 
socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140415-104517-rAfzfDef,server,nowait
 \
 -device isa-serial,chardev=serial_id_serial0  \
 -chardev 
socket,id=seabioslog_id_20140415-104517-rAfzfDef,path=/tmp/seabios-20140415-104517-rAfzfDef,server,nowait
 \
 -device 
isa-debugcon,chardev=seabioslog_id_20140415-104517-rAfzfDef,iobase=0x402 \
 -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \
 -drive 
id=drive_image1,if=none,file=/home/dgilbert/virt-test/shared/data/images/jeos-19-64.qcow2
 \
 -device 
virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=04 \
 -device 
virtio-net-pci,mac=9a:18:19:1a:1b:1c,id=idz0uWFP,vectors=4,netdev=idMwmdMc,bus=pci.0,addr=05
  \
 -netdev user,id=idMwmdMc,hostfwd=tcp::5000-:22  \
 -m 1024  \
 -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
 -cpu 'SandyBridge' \
 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
 -vnc :1  \
 -rtc base=utc,clock=host,driftfix=none  \
 -boot order=cdn,once=c,menu=off \
 -enable-kvm

Dave


OK biggest bummer it works fine with all freshly started VMs. No idea why.

So i'll try with paolos idea migrating old VMs.




Thanks!

Stefan



Dave
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

Re: [Qemu-devel] Migration from older Qemu to Qemu 2.0.0 does not work

[Stefan Priebe]

)

Am 09.05.2014 19:05, schrieb Paolo Bonzini:

Il 09/05/2014 15:13, Stefan Priebe - Profihost AG ha scritto:


I see no output at the monitor of Qemu 2.0.

# migrate -d tcp:a.b.c.d:6000

# info migrate
capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on
zero-blocks: off
Migration status: failed
total time: 0 milliseconds

The target machine is still running at this point with no output.


Anything on its stdout?


Sadly no.

But far more complex than i thought. I'm seeing this problem only on 
machines originally started under qemu 1.5 ten migrated to qemu 1.7 with 
machine type=pc440fc-1.5 . Only those do not work. But they also do not 
work migrating them to qemu 1.7.1.


So let's ignore the whole fact and thanks for your bisect ideas. I will 
just shut them down all and start them with a fresh qemu 2.0.



Another test you could do, in addition to changing the devices, is this:

1) try with commit 6141f3bd6904df7cf9519c6444a14a608b9874c4 on the
destination (the next one caused a migration problem that was fixed
later).  If it passes, go to step 1a.  If it fails, go to step 2.

1a) try with commit c01a71c1a56fa27f43449ff59e5d03b2483658a2.  If it
passes, go to step 1b.  If it fails, report it here.

1b) bisect between v2.0.0 (known-bad) and
c01a71c1a56fa27f43449ff59e5d03b2483658a2 (known-good) to find the
culprit.  Report results.

2) change the source to v1.7.0 and bisect between v1.7.0 and
6141f3bd6904df7cf9519c6444a14a608b9874c4.  Report results.

Thanks!

Paolo

Re: [Qemu-devel] [PATCH 00/35] pc: ACPI memory hotplug

[Stefan Priebe - Profihost AG]

Hello Igor,

while testing your patchset i came to a very stupid problem.

I wanted to test migration and it cames out that the migration works
fine after plugging in memory only if i run the target VM without the
-daemonize option.

If i enable the -daemonize option the target vm tries to read from non
readable memory.

proc maps shows:
7f9334021000-7f933800 ---p  00:00 0

where it tries to read from.

Also the memory layout is different in daemonize mode than in non
daemonize mode.

Stefan

Am 04.04.2014 15:36, schrieb Igor Mammedov:
 What's new since v7:
 
 * Per Andreas' suggestion dropped DIMMBus concept.
 
 * Added hotplug binding for bus-less devices
 
 * DIMM device is split to backend and frontend. Therefore following
   command/options were added for supporting it:
 
   For memory-ram backend:
   CLI: -object-add memory-ram,
   with options: 'id' and 'size'
   For dimm frontend:
   option size became readonly, pulling it's size from attached backend
   added option memdev for specifying backend by 'id'
 
 * dropped support for 32 bit guests
 
 * failed hotplug action doesn't consume 1 slot anymore
 
 * vaious fixes adressing reviewer's comments most of them in ACPI part
 ---
 
 This series allows to hotplug 'arbitrary' DIMM devices specifying size,
 NUMA node mapping (guest side), slot and address where to map it, at runtime.
 
 Due to ACPI limitation there is need to specify a number of possible
 DIMM devices. For this task -m option was extended to support
 following format:
 
   -m [mem=]RamSize[,slots=N,maxmem=M]
 
 To allow memory hotplug user must specify a pair of additional parameters:
 'slots' - number of possible increments
 'maxmem' - max possible total memory size QEMU is allowed to use,
including RamSize.
 
 minimal monitor command syntax to hotplug DIMM device:
 
   object_add memory-ram,id=memX,size=1G
   device_add dimm,id=dimmX,memdev=memX
 
 DIMM device provides following properties that could be used with
 device_add / -device to alter default behavior:
 
   id- unique string identifying device [mandatory]
   slot  - number in range [0-slots) [optional], if not specified
   the first free slot is used
   node  - NUMA node id [optional] (default: 0)
   size  - amount of memory to add, readonly derived from backing memdev
   start - guest's physical address where to plug DIMM [optional],
   if not specified the first gap in hotplug memory region
   that fits DIMM is used
 
  -device option could be used for adding potentially hotunplugable DIMMs
 and also for specifying hotplugged DIMMs in migration case.
 
 Tested guests:
  - RHEL 6x64
  - Windows 2012DCx64
  - Windows 2008DCx64
 
 Known limitations/bugs/TODOs:
  - hot-remove is not supported, yet
  - max number of supported DIMM devices 255 (due to ACPI object name
limit), could be increased creating several containers and putting
DIMMs there. (exercise for future) 
  - e820 table doesn't include DIMM devices added with -device /
(or after reboot devices added with device_add)
  - Windows 2008 remembers DIMM configuration, so if DIMM with other
start/size is added into the same slot, it refuses to use it insisting
on old mapping.
 
 QEMU git tree for testing is available at:
   https://github.com/imammedo/qemu/commits/memory-hotplug-v8
 
 Example QEMU cmd line:
   qemu-system-x86_64 -enable-kvm -monitor unix:/tmp/mon,server,nowait \ 
  -m 4096,slots=4,maxmem=8G guest.img
 
 PS:
   Windows guest requires SRAT table for hotplug to work so add an extra 
 option:
-numa node
   to QEMU command line.
 
 
 Igor Mammedov (34):
   vl: convert -m to QemuOpts
   object_add: allow completion handler to get canonical path
   add memdev backend infrastructure
   vl.c: extend -m option to support options for memory hotplug
   add pc-{i440fx,q35}-2.1 machine types
   pc: create custom generic PC machine type
   qdev: hotplug for buss-less devices
   qdev: expose DeviceState.hotplugged field as a property
   dimm: implement dimm device abstraction
   memory: add memory_region_is_mapped() API
   dimm: do not allow to set already busy memdev
   pc: initialize memory hotplug address space
   pc: exit QEMU if slots  256
   pc: add 'etc/reserved-memory-end' fw_cfg interface for SeaBIOS
   pc: add memory hotplug handler to PC_MACHINE
   dimm: add busy address check and address auto-allocation
   dimm: add busy slot check and slot auto-allocation
   acpi: rename cpu_hotplug_defs.h to acpi_defs.h
   acpi: memory hotplug ACPI hardware implementation
   trace: add acpi memory hotplug IO region events
   trace: add DIMM slot  address allocation for target-i386
   acpi:piix4: make plug/unlug callbacks generic
   acpi:piix4: add memory hotplug handling
   pc: ich9 lpc: make it work with global/compat properties
   acpi:ich9: add memory hotplug handling
   pc: migrate piix4  ich9 MemHotplugState
   pc: propagate memory hotplug event to ACPI device

[Qemu-devel] multiple USB Serial devices qemu load

[Stefan Priebe - Profihost AG]

Hello,

i've 4 USB serial devices and one HID device i want to pass to a guest.

The passing itself works fine but while the guest has 0 load or cpu
usage the qemu process itself has around 40% cpu usage on a single 3,2
ghz E3 xeon.

I already tried xhci but it doesn't change anything. Also the latency of
the usb devices is very high and fluctuate. Qemu Version 1.7.1

Inside the guest:
# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
|__ Port 1: Dev 2, If 0, Class=HID, Driver=usbhid, 480M
|__ Port 2: Dev 3, If 0, Class=HID, Driver=usbfs, 12M
|__ Port 3: Dev 4, If 0, Class=vend., Driver=ftdi_sio, 12M
|__ Port 4: Dev 5, If 0, Class=hub, Driver=hub/8p, 12M
|__ Port 1: Dev 6, If 0, Class=vend., Driver=ftdi_sio, 12M
|__ Port 2: Dev 7, If 0, Class=comm., Driver=cdc_acm, 12M
|__ Port 2: Dev 7, If 1, Class=data, Driver=cdc_acm, 12M
|__ Port 3: Dev 8, If 0, Class=vend., Driver=ftdi_sio, 12M

Qremu command line:

qemu -pidfile /var/run/qemu-server/102.pid -daemonize -name test -smp
sockets=1,cores=1 -nodefaults -boot
menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu
kvm64,+lahf_lm,+x2apic,+sep -k de -m 1024 -device
nec-usb-xhci,id=xhci,bus=pci.0,addr=0x1.0x2 -device
usb-tablet,id=tablet,bus=xhci.0,port=1 -device
usb-host,hostbus=2,hostport=1.2 -device usb-host,hostbus=2,hostport=1.3
-device usb-host,hostbus=2,hostport=1.4 -device
usb-host,hostbus=2,hostport=1.1.1 -device
usb-host,hostbus=2,hostport=1.1.4 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -device
virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive
file=/ssdstor/images/102/vm-102-disk-1.qcow2,if=none,id=drive-scsi0,format=qcow2,cache=writeback,discard=on,aio=native
-device
scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0
-netdev
type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,vhost=on
-device
virtio-net-pci,mac=3A:53:02:E3:76:59,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300

Does anybody have an idea how to lower the CPU usage?

Greets,
Stefan

Re: [Qemu-devel] memory allocation of migration changed?

[Stefan Priebe]

Am 24.02.2014 17:13, schrieb Eric Blake:

On 02/24/2014 08:00 AM, Stefan Hajnoczi wrote:


What is the right way to check for enough free memory and memory
usage of a specific vm?


I would approach it in terms of guest RAM allocation plus QEMU overhead:

   host_ram = num_guests * guest_ram_size + num_guests * qemu_overhead

The qemu_overhead is the question mark.  It depends on the QEMU features
the guests have enabled and number of devices.

QEMU also does not have a strict policy on predictable memory
consumption, which makes it hard to give a formula for it.


In fact, at one point libvirt tried to put an automatic cap on the
memory usable by qemu by multiplying RAM size and accounting for a
margin of overhead, but no matter what heuristics we tried, we still got
complaints from users that their guests were killed when they ran out of
memory, and so we ended up reverting the automatic limits from libvirt.
  (You can still enforce a limit as an end user, although the libvirt
documentation no longer recommends attempting that, for as long as the
qemu allocation remains unpredictable.)



Might something like:
vm.overcommit_ratio=100
vm.overcommit_memory=2

help? So that the migration might not happen but that's better than 
killing a random process?


Stefan

Re: [Qemu-devel] memory allocation of migration changed?

[Stefan Priebe]

Am 14.02.2014 15:59, schrieb Stefan Hajnoczi:

On Tue, Feb 11, 2014 at 07:32:46PM +0100, Stefan Priebe wrote:

Am 11.02.2014 17:22, schrieb Peter Lieven:




Am 11.02.2014 um 16:44 schrieb Stefan Hajnoczi stefa...@gmail.com:

On Tue, Feb 11, 2014 at 3:54 PM, Stefan Priebe - Profihost AG
s.pri...@profihost.ag wrote:

in the past (Qemu 1.5) a migration failed if there was not enogh memory
on the target host available directly at the beginning.

Now with Qemu 1.7 i've seen succeeded migrations but the kernel OOM
memory killer killing qemu processes. So the migration seems to takes
place without having anough memory on the target machine?


How much memory is the guest configured with?  How much memory does
the host have?

I wonder if there are zero pages that can be migrated almost for
free and the destination host doesn't touch.  When they are touched
for the first time after migration handover, they need to be allocated
on the destination host.  This can lead to OOM if you overcommitted
memory.

Can you reproduce the OOM reliably?  It should be possible to debug it
and figure out whether it's just bad luck or a true regression.

Stefan


Kernel Version would also be interesting as well as thp and ksm settings.


Kernel Host: 3.10.26

What's thp / ksm? how to get those settings?


Transparent Huge Pages

# cat /sys/kernel/mm/transparent_hugepage/enabled

Kernel Samepage Merging

# cat /sys/kernel/mm/ksm/run



# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never


# cat /sys/kernel/mm/ksm/run
1




Stefan

Re: [Qemu-devel] memory allocation of migration changed?

[Stefan Priebe]

Am 14.02.2014 16:03, schrieb Stefan Hajnoczi:

On Tue, Feb 11, 2014 at 07:30:54PM +0100, Stefan Priebe wrote:

Am 11.02.2014 16:44, schrieb Stefan Hajnoczi:

On Tue, Feb 11, 2014 at 3:54 PM, Stefan Priebe - Profihost AG
s.pri...@profihost.ag wrote:

in the past (Qemu 1.5) a migration failed if there was not enogh memory
on the target host available directly at the beginning.

Now with Qemu 1.7 i've seen succeeded migrations but the kernel OOM
memory killer killing qemu processes. So the migration seems to takes
place without having anough memory on the target machine?


How much memory is the guest configured with?  How much memory does
the host have?


Guest: 48GB
Host: 192GB


I wonder if there are zero pages that can be migrated almost for
free and the destination host doesn't touch.  When they are touched
for the first time after migration handover, they need to be allocated
on the destination host.  This can lead to OOM if you overcommitted
memory.


In the past the migration failed immediatly with exit code 255.


Can you reproduce the OOM reliably?  It should be possible to debug it
and figure out whether it's just bad luck or a true regression.


So there is no known patch changing this behaviour?

What is about those?
fc1c4a5d32e15a4c40c47945da85ef9c1e0c1b54
211ea74022f51164a7729030b28eec90b6c99a08
f1c72795af573b24a7da5eb52375c9aba8a37972


Yes, that's what I was referring to when I mentioned zero pages.

The problem might just be that the destination host didn't have enough
free memory.  Migration succeeded due to memory overcommit on the host,
but quickly ran out of memory after handover.  The quick answer there is
to reconsider your overcommitting memory and also checking memory
availability before live migrating.


Yes that makes sense in the past it was just working due to a failing 
qemu process.


What is the right way to check for enough free memory and memory usage 
of a specific vm?


Stefan

Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry

[Stefan Priebe]

Am 13.02.2014 21:06, schrieb Dr. David Alan Gilbert:

* Stefan Priebe (s.pri...@profihost.ag) wrote:

Am 10.02.2014 17:07, schrieb Dr. David Alan Gilbert:

* Stefan Priebe (s.pri...@profihost.ag) wrote:

i could fix it by explicitly disable xbzrle - it seems its
automatically on if i do not set the migration caps to false.

So it seems to be a xbzrle bug.


Stefan can you give me some more info on your hardware and
migration setup;   that stressapptest (which is a really nice
find!) really batters the memory and it means the migration
isn't converging for me, so I'm curious what your setup is.


That one is devlopment by google and known to me since a few years.
Google has detected that memtest and co are not good enough to
stress test memory.


Hi Stefan,
   I've just posted a patch to qemu-devel that fixes two bugs that
we found; I've only tried a small stressapptest run and it seems
to survive with them (where it didn't before);  you might like to try
it if you're up for rebuilding qemu.

It's the one entitled ' [PATCH] Fix two XBZRLE corruption issues'


Thanks!

Really would love to try them but nor google nor myself can find them.

http://osdir.com/ml/qemu-devel/2014-02/

Stefan



I'll try and get a larger run done myself, but I'd be interested to
hear if it fixes it for you (or anyone else who hit the problem).

Dave
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK