RE: [PATCH v1 0/2] TM field check failed

[Zhang, Qi1]

Hello, everyone

May I know you have comments on this patch set? Thank you!

BRs
Qi Zhang
> -Original Message-
> From: Zhang, Qi1 
> Sent: Tuesday, October 8, 2019 10:35 AM
> To: qemu-devel@nongnu.org
> Cc: m...@redhat.com; marcel.apfelb...@gmail.com; pbonz...@redhat.com;
> r...@twiddle.net; ehabk...@redhat.com; Zhang, Qi1 ;
> Qi, Yadong 
> Subject: [PATCH v1 0/2] TM field check failed
> 
> From: "Zhang, Qi" 
> 
> spilt the reserved fields arrays and remove TM field from reserved bits
> 
> Changelog V1:
>   add descriptons
> 
> Zhang, Qi (2):
>   intel_iommu: split the resevred fields arrays into two ones
>   intel_iommu: TM field should not be in reserved bits
> 
>  hw/i386/intel_iommu.c  | 35 --
>  hw/i386/intel_iommu_internal.h | 17 +
>  2 files changed, 34 insertions(+), 18 deletions(-)
> 
> --
> 2.20.1

[PATCH] target/i386/kvm: Add Hyper-V direct tlb flush support

[lantianyu1986]

From: Tianyu Lan 

Hyper-V direct tlb flush targets KVM on Hyper-V guest.
Enable direct TLB flush for its guests meaning that TLB
flush hypercalls are handled by Level 0 hypervisor (Hyper-V)
bypassing KVM in Level 1. Due to the different ABI for hypercall
parameters between Hyper-V and KVM, KVM capabilities should be
hidden when enable Hyper-V direct tlb flush otherwise KVM
hypercalls may be intercepted by Hyper-V. Add new parameter
"hv-direct-tlbflush". Check expose_kvm and Hyper-V tlb flush
capability status before enabling the feature.

Signed-off-by: Tianyu Lan 
---
 docs/hyperv.txt   | 12 
 linux-headers/linux/kvm.h |  1 +
 target/i386/cpu.c |  2 ++
 target/i386/cpu.h |  1 +
 target/i386/kvm.c | 21 +
 5 files changed, 37 insertions(+)

diff --git a/docs/hyperv.txt b/docs/hyperv.txt
index 8fdf25c829..ceab8c21fe 100644
--- a/docs/hyperv.txt
+++ b/docs/hyperv.txt
@@ -184,6 +184,18 @@ enabled.
 
 Requires: hv-vpindex, hv-synic, hv-time, hv-stimer
 
+3.18. hv-direct-tlbflush
+===
+The enlightenment targets KVM on Hyper-V guest. Enable direct TLB flush for
+its guests meaning that TLB flush hypercalls are handled by Level 0 hypervisor
+(Hyper-V) bypassing KVM in Level 1. Due to the different ABI for hypercall
+parameters between Hyper-V and KVM, enabling this capability effectively
+disables all hypercall handling by KVM (as some KVM hypercall may be mistakenly
+treated as TLB flush hypercalls by Hyper-V). So kvm capability should not show
+to guest when enable this capability. If not, user will fail to enable this
+capability.
+
+Requires: hv-tlbflush, -kvm
 
 4. Development features
 
diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 18892d6541..923fb33a01 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -995,6 +995,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_ARM_SVE 170
 #define KVM_CAP_ARM_PTRAUTH_ADDRESS 171
 #define KVM_CAP_ARM_PTRAUTH_GENERIC 172
+#define KVM_CAP_HYPERV_DIRECT_TLBFLUSH 174
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 44f1bbdcac..7bc7fee512 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -6156,6 +6156,8 @@ static Property x86_cpu_properties[] = {
   HYPERV_FEAT_IPI, 0),
 DEFINE_PROP_BIT64("hv-stimer-direct", X86CPU, hyperv_features,
   HYPERV_FEAT_STIMER_DIRECT, 0),
+DEFINE_PROP_BIT64("hv-direct-tlbflush", X86CPU, hyperv_features,
+  HYPERV_FEAT_DIRECT_TLBFLUSH, 0),
 DEFINE_PROP_BOOL("hv-passthrough", X86CPU, hyperv_passthrough, false),
 
 DEFINE_PROP_BOOL("check", X86CPU, check_cpuid, true),
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index eaa5395aa5..3cb105f7d6 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -907,6 +907,7 @@ typedef uint64_t FeatureWordArray[FEATURE_WORDS];
 #define HYPERV_FEAT_EVMCS   12
 #define HYPERV_FEAT_IPI 13
 #define HYPERV_FEAT_STIMER_DIRECT   14
+#define HYPERV_FEAT_DIRECT_TLBFLUSH 15
 
 #ifndef HYPERV_SPINLOCK_NEVER_RETRY
 #define HYPERV_SPINLOCK_NEVER_RETRY 0x
diff --git a/target/i386/kvm.c b/target/i386/kvm.c
index 11b9c854b5..8e999dbcf1 100644
--- a/target/i386/kvm.c
+++ b/target/i386/kvm.c
@@ -1235,6 +1235,27 @@ static int hyperv_handle_properties(CPUState *cs,
 r |= 1;
 }
 
+if (hyperv_feat_enabled(cpu, HYPERV_FEAT_DIRECT_TLBFLUSH)) {
+if (!kvm_check_extension(cs->kvm_state,
+KVM_CAP_HYPERV_DIRECT_TLBFLUSH)) {
+fprintf(stderr,
+"Kernel doesn't support Hyper-V direct tlbflush.\n");
+r = -ENOSYS;
+goto free;
+}
+
+if (cpu->expose_kvm ||
+!hyperv_feat_enabled(cpu, HYPERV_FEAT_TLBFLUSH)) {
+fprintf(stderr, "Hyper-V direct tlbflush requires Hyper-V %s"
+" and not expose KVM.\n",
+kvm_hyperv_properties[HYPERV_FEAT_TLBFLUSH].desc);
+r = -ENOSYS;
+goto free;
+}
+
+kvm_vcpu_enable_cap(cs, KVM_CAP_HYPERV_DIRECT_TLBFLUSH, 0, 0);
+}
+
 /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
 env->features[FEAT_HYPERV_EDX] |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;
 
-- 
2.14.5

Re: [PATCH v4 2/2] i386: Add support to get/set/migrate Intel Processor Trace feature

[Eduardo Habkost]

On Mon, Mar 05, 2018 at 12:48:36AM +0800, Luwei Kang wrote:
> From: Chao Peng 
> 
> Add Intel Processor Trace related definition. It also add
> corresponding part to kvm_get/set_msr and vmstate.
> 
> Signed-off-by: Chao Peng 
> Signed-off-by: Luwei Kang 
[...]
> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
> index f9f4cd1..097c953 100644
> --- a/target/i386/kvm.c
> +++ b/target/i386/kvm.c
> @@ -1811,6 +1811,25 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
>  kvm_msr_entry_add(cpu, MSR_MTRRphysMask(i), mask);
>  }
>  }
> +if (env->features[FEAT_7_0_EBX] & CPUID_7_0_EBX_INTEL_PT) {
> +int addr_num = kvm_arch_get_supported_cpuid(kvm_state,
> +0x14, 1, R_EAX) & 0x7;
> +
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_CTL,
> +env->msr_rtit_ctrl);
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_STATUS,
> +env->msr_rtit_status);
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_OUTPUT_BASE,
> +env->msr_rtit_output_base);

This causes the following crash on some hosts:

  qemu-system-x86_64: error: failed to set MSR 0x560 to 0x0
  qemu-system-x86_64: target/i386/kvm.c:2673: kvm_put_msrs: Assertion `ret == 
cpu->kvm_msr_buf->nmsrs' failed.

Checking for CPUID_7_0_EBX_INTEL_PT is not enough: KVM has
additional conditions that might prevent writing to this MSR
(PT_CAP_topa_output && PT_CAP_single_range_output).  This causes
QEMU to crash if some of the conditions aren't met.

Writing and reading this MSR (and the ones below) need to be
conditional on KVM_GET_MSR_INDEX_LIST.


> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_OUTPUT_MASK,
> +env->msr_rtit_output_mask);
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_CR3_MATCH,
> +env->msr_rtit_cr3_match);
> +for (i = 0; i < addr_num; i++) {
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_ADDR0_A + i,
> +env->msr_rtit_addrs[i]);
> +}
> +}
>  
>  /* Note: MSR_IA32_FEATURE_CONTROL is written separately, see
>   *   kvm_put_msr_feature_control. */
> @@ -2124,6 +2143,20 @@ static int kvm_get_msrs(X86CPU *cpu)
>  }
>  }
>  
> +if (env->features[FEAT_7_0_EBX] & CPUID_7_0_EBX_INTEL_PT) {
> +int addr_num =
> +kvm_arch_get_supported_cpuid(kvm_state, 0x14, 1, R_EAX) & 0x7;
> +
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_CTL, 0);
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_STATUS, 0);
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_OUTPUT_BASE, 0);
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_OUTPUT_MASK, 0);
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_CR3_MATCH, 0);
> +for (i = 0; i < addr_num; i++) {
> +kvm_msr_entry_add(cpu, MSR_IA32_RTIT_ADDR0_A + i, 0);
> +}
> +}
> +
>  ret = kvm_vcpu_ioctl(CPU(cpu), KVM_GET_MSRS, cpu->kvm_msr_buf);
>  if (ret < 0) {
>  return ret;
> @@ -2364,6 +2397,24 @@ static int kvm_get_msrs(X86CPU *cpu)
>  case MSR_IA32_SPEC_CTRL:
>  env->spec_ctrl = msrs[i].data;
>  break;
> +case MSR_IA32_RTIT_CTL:
> +env->msr_rtit_ctrl = msrs[i].data;
> +break;
> +case MSR_IA32_RTIT_STATUS:
> +env->msr_rtit_status = msrs[i].data;
> +break;
> +case MSR_IA32_RTIT_OUTPUT_BASE:
> +env->msr_rtit_output_base = msrs[i].data;
> +break;
> +case MSR_IA32_RTIT_OUTPUT_MASK:
> +env->msr_rtit_output_mask = msrs[i].data;
> +break;
> +case MSR_IA32_RTIT_CR3_MATCH:
> +env->msr_rtit_cr3_match = msrs[i].data;
> +break;
> +case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B:
> +env->msr_rtit_addrs[index - MSR_IA32_RTIT_ADDR0_A] = 
> msrs[i].data;
> +break;
>  }
>  }
>  
> diff --git a/target/i386/machine.c b/target/i386/machine.c
> index 361c05a..c05fe6f 100644
> --- a/target/i386/machine.c
> +++ b/target/i386/machine.c
> @@ -837,6 +837,43 @@ static const VMStateDescription vmstate_spec_ctrl = {
>  }
>  };
>  
> +static bool intel_pt_enable_needed(void *opaque)
> +{
> +X86CPU *cpu = opaque;
> +CPUX86State *env = >env;
> +int i;
> +
> +if (env->msr_rtit_ctrl || env->msr_rtit_status ||
> +env->msr_rtit_output_base || env->msr_rtit_output_mask ||
> +env->msr_rtit_cr3_match) {
> +return true;
> +}
> +
> +for (i = 0; i < MAX_RTIT_ADDRS; i++) {
> +if (env->msr_rtit_addrs[i]) {
> +return true;
> +}
> +}
> +
> +return false;
> +}
> +
> +static const VMStateDescription vmstate_msr_intel_pt = {
> +.name = "cpu/intel_pt",
> +.version_id = 1,
> +.minimum_version_id = 1,
> +.needed = intel_pt_enable_needed,
> 

Re: [PATCH v12 09/11] hmat acpi: Build System Locality Latency and Bandwidth Information Structure(s)

[Tao Xu]

On 10/11/2019 10:08 PM, Igor Mammedov wrote:

On Thu, 10 Oct 2019 14:53:56 +0800
Tao Xu  wrote:


On 10/3/2019 10:41 PM, Igor Mammedov wrote:

On Fri, 20 Sep 2019 15:43:47 +0800
Tao Xu  wrote:
   

From: Liu Jingqi 

This structure describes the memory access latency and bandwidth
information from various memory access initiator proximity domains.
The latency and bandwidth numbers represented in this structure
correspond to rated latency and bandwidth for the platform.
The software could use this information as hint for optimization.

Signed-off-by: Liu Jingqi 
Signed-off-by: Tao Xu 
---

Changes in v12:
  - Fix a bug that if HMAT is enabled and without hmat-lb setting,
QEMU will crash. (reported by Danmei Wei)

Changes in v11:
  - Calculate base in build_hmat_lb().
---
   hw/acpi/hmat.c | 126 -
   hw/acpi/hmat.h |   2 +
   2 files changed, 127 insertions(+), 1 deletion(-)

diff --git a/hw/acpi/hmat.c b/hw/acpi/hmat.c
index 1368fce7ee..e7be849581 100644
--- a/hw/acpi/hmat.c
+++ b/hw/acpi/hmat.c
@@ -27,6 +27,7 @@
   #include "qemu/osdep.h"
   #include "sysemu/numa.h"
   #include "hw/acpi/hmat.h"
+#include "qemu/error-report.h"
   
   /*

* ACPI 6.3:
@@ -67,11 +68,105 @@ static void build_hmat_mpda(GArray *table_data, uint16_t 
flags, int initiator,
   build_append_int_noprefix(table_data, 0, 8);
   }
   
+static bool entry_overflow(uint64_t *lb_data, uint64_t base, int len)

+{
+int i;
+
+for (i = 0; i < len; i++) {
+if (lb_data[i] / base >= UINT16_MAX) {
+return true;
+}
+}
+
+return false;
+}

I suggest to do this check at CLI parsing time
   

+/*
+ * ACPI 6.3: 5.2.27.4 System Locality Latency and Bandwidth Information
+ * Structure: Table 5-146
+ */
+static void build_hmat_lb(GArray *table_data, HMAT_LB_Info *hmat_lb,
+  uint32_t num_initiator, uint32_t num_target,
+  uint32_t *initiator_list, int type)
+{
+uint8_t mask = 0x0f;
+uint32_t s = num_initiator;
+uint32_t t = num_target;

drop this locals and use arguments directly
   

+uint64_t base = 1;
+uint64_t *lb_data;
+int i, unit;
+
+/* Type */
+build_append_int_noprefix(table_data, 1, 2);
+/* Reserved */
+build_append_int_noprefix(table_data, 0, 2);
+/* Length */
+build_append_int_noprefix(table_data, 32 + 4 * s + 4 * t + 2 * s * t, 4);

   
to me above looks like /dev/random output, absolutely unreadable.
Suggest to use local var (like: lb_length) for expression with comments
beside magic numbers.
   

+/* Flags: Bits [3:0] Memory Hierarchy, Bits[7:4] Reserved */
+build_append_int_noprefix(table_data, hmat_lb->hierarchy & mask, 1);


why do you need to use mask here?
   

Because Bits[7:4] Reserved, so I use mask to keep it reserved.


these bits are not user provided and set to 0, if they get set it's
programming error and instead of masking problem out QEMU should abort,
I suggest replace masking with assert(!foo>>x).




+/* Data Type */
+build_append_int_noprefix(table_data, hmat_lb->data_type, 1);


Isn't hmat_lb->data_type and passed argument 'type' the same?
   

Yes, I will drop 'type'.
   

+/* Reserved */
+build_append_int_noprefix(table_data, 0, 2);
+/* Number of Initiator Proximity Domains (s) */
+build_append_int_noprefix(table_data, s, 4);
+/* Number of Target Proximity Domains (t) */
+build_append_int_noprefix(table_data, t, 4);
+/* Reserved */
+build_append_int_noprefix(table_data, 0, 4);
+
+if (HMAT_IS_LATENCY(type)) {
+unit = 1000;
+lb_data = hmat_lb->latency;
+} else {
+unit = 1024;
+lb_data = hmat_lb->bandwidth;
+}
+
+while (entry_overflow(lb_data, base, s * t)) {
+for (i = 0; i < s * t; i++) {
+if (!QEMU_IS_ALIGNED(lb_data[i], unit * base)) {
+error_report("Invalid latency/bandwidth input, all "
+"latencies/bandwidths should be specified in the same units.");
+exit(1);
+}
+}
+base *= unit;
+}

Can you clarify what you are trying to check here?
   

This part I use entry_overflow() to check if uint16 can store entry. If
can't store and the entries matrix can be divisible by unit * base, then
base will be unit * base.

For example, if lb_data[i] are 1048576(1TB/s) and 1024(1GB/s), unit is
1024, so 1048576 is bigger than UINT16_MAX, and can be divisible by 1024
* 1, so base is 1024 and entries are 1024 and 1 (see entry =
hmat_lb->latency[i] / base;). The benefit is even user input different
unit(TB/s vs GB/s), we can still store the data as far as possible.


Is it possible instead of doing multiple iterations over lb_data
until it finds valid base, just go over lb_data once to find MIN/MAX
and then calculate base using it. Error out with max/min offending
values if it's not 

[PATCH v3] target/i386: Add Snowridge-v2 (no MPX) CPU model

[Xiaoyao Li]

Add new version of Snowridge CPU model that removes MPX feature.

MPX support is being phased out by Intel. GCC has dropped it, Linux kernel
and KVM are also going to do that in the future.

Signed-off-by: Xiaoyao Li 
---
Changes in v3:
- Remove the .alias field (ehabkost)
 
Changes in v2:
- Use CPU model versioning mechanism instead of machine-type compat
---
 target/i386/cpu.c | 12 
 1 file changed, 12 insertions(+)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 44f1bbdcac76..6685f803da02 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -2793,6 +2793,18 @@ static X86CPUDefinition builtin_x86_defs[] = {
 CPUID_6_EAX_ARAT,
 .xlevel = 0x8008,
 .model_id = "Intel Atom Processor (SnowRidge)",
+.versions = (X86CPUVersionDefinition[]) {
+{ .version = 1 },
+{
+.version = 2,
+.props = (PropValue[]) {
+{ "mpx", "off" },
+{ "model-id", "Intel Atom Processor (Snowridge, no MPX)" },
+{ /* end of list */ },
+},
+},
+{ /* end of list */ },
+},
 },
 {
 .name = "KnightsMill",
-- 
2.19.1

Re: [RFC v5 000/126] error: auto propagated local_err

[no-reply]

Patchew URL: 
https://patchew.org/QEMU/20191011160552.22907-1-vsement...@virtuozzo.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [RFC v5 000/126] error: auto propagated local_err
Type: series
Message-id: 20191011160552.22907-1-vsement...@virtuozzo.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
319b206 util/qemu-config.c: introduce ERRP_AUTO_PROPAGATE
3ee6567 tests/test-image-locking.c: introduce ERRP_AUTO_PROPAGATE
2e4f371 target/tilegx/cpu.c: introduce ERRP_AUTO_PROPAGATE
5f766d7 memory_mapping.c: introduce ERRP_AUTO_PROPAGATE
8969c6f iothread.c: introduce ERRP_AUTO_PROPAGATE
1534a92 hw/sd/ssi-sd.c: introduce ERRP_AUTO_PROPAGATE
b5b1e0a hw/cpu/core.c: introduce ERRP_AUTO_PROPAGATE
dd3f310 hw/core/bus.c: introduce ERRP_AUTO_PROPAGATE
21ee5f8 PVRDMA: introduce ERRP_AUTO_PROPAGATE
84365ab Replication: introduce ERRP_AUTO_PROPAGATE
225bbc5 vvfat: introduce ERRP_AUTO_PROPAGATE
2da58d6 vpc: introduce ERRP_AUTO_PROPAGATE
e7fb0b9 blkdebug: introduce ERRP_AUTO_PROPAGATE
39d367e qcow: introduce ERRP_AUTO_PROPAGATE
1b7546c qcow2: introduce ERRP_AUTO_PROPAGATE
c0352a1 raw: introduce ERRP_AUTO_PROPAGATE
612aebf qed: introduce ERRP_AUTO_PROPAGATE
155f6ea parallels: introduce ERRP_AUTO_PROPAGATE
5d3c3fd blkverify: introduce ERRP_AUTO_PROPAGATE
36132e6 blklogwrites: introduce ERRP_AUTO_PROPAGATE
785cfb4 Quorum: introduce ERRP_AUTO_PROPAGATE
5acba63 Bootdevice: introduce ERRP_AUTO_PROPAGATE
0e0d3fb NVMe Block Driver: introduce ERRP_AUTO_PROPAGATE
1cf6911 GLUSTER: introduce ERRP_AUTO_PROPAGATE
cb4c8f7 CURL: introduce ERRP_AUTO_PROPAGATE
4e32493 SSH: introduce ERRP_AUTO_PROPAGATE
abbecd4 NFS: introduce ERRP_AUTO_PROPAGATE
f4f937d nbd: introduce ERRP_AUTO_PROPAGATE
7b638b3 iSCSI: introduce ERRP_AUTO_PROPAGATE
4727ffc VDI: introduce ERRP_AUTO_PROPAGATE
2c03145 VHDX: introduce ERRP_AUTO_PROPAGATE
7916a87 Sheepdog: introduce ERRP_AUTO_PROPAGATE
db15fdd RBD: introduce ERRP_AUTO_PROPAGATE
05ea2cf VMDK: introduce ERRP_AUTO_PROPAGATE
ff11144 Record/replay: introduce ERRP_AUTO_PROPAGATE
a95fab2 colo: introduce ERRP_AUTO_PROPAGATE
be71202 Sockets: introduce ERRP_AUTO_PROPAGATE
69a59b0 I/O Channels: introduce ERRP_AUTO_PROPAGATE
e4f56f3 Cryptography: introduce ERRP_AUTO_PROPAGATE
4f5f412 Migration: introduce ERRP_AUTO_PROPAGATE
985da1a TPM: introduce ERRP_AUTO_PROPAGATE
b19cdab Tracing: introduce ERRP_AUTO_PROPAGATE
3113fc7 SLIRP: introduce ERRP_AUTO_PROPAGATE
51e2f48 QMP: introduce ERRP_AUTO_PROPAGATE
1c0c827 QOM: introduce ERRP_AUTO_PROPAGATE
fc0eec4 qga: introduce ERRP_AUTO_PROPAGATE
af16041 QAPI: introduce ERRP_AUTO_PROPAGATE
21ed21e cryptodev: introduce ERRP_AUTO_PROPAGATE
7ab6e12 hostmem: introduce ERRP_AUTO_PROPAGATE
994c02c net: introduce ERRP_AUTO_PROPAGATE
26fe9a4 Human Monitor (HMP): introduce ERRP_AUTO_PROPAGATE
82b7f8b Main loop: introduce ERRP_AUTO_PROPAGATE
863100d Graphics: introduce ERRP_AUTO_PROPAGATE
45a8d41 SPICE: introduce ERRP_AUTO_PROPAGATE
6d967ec Memory API: introduce ERRP_AUTO_PROPAGATE
5645325 Dump: introduce ERRP_AUTO_PROPAGATE
6d795b4 cmdline: introduce ERRP_AUTO_PROPAGATE
5fceaa3 chardev: introduce ERRP_AUTO_PROPAGATE
d551bda scsi: introduce ERRP_AUTO_PROPAGATE
cc3d83e block: introduce ERRP_AUTO_PROPAGATE
75b948b Audio: introduce ERRP_AUTO_PROPAGATE
c3fee2f XIVE: introduce ERRP_AUTO_PROPAGATE
42ba3e1 fw_cfg: introduce ERRP_AUTO_PROPAGATE
90c4efa virtio-gpu: introduce ERRP_AUTO_PROPAGATE
4db3f47 eepro100: introduce ERRP_AUTO_PROPAGATE
d7634f4 NVDIMM: introduce ERRP_AUTO_PROPAGATE
706ee21 megasas: introduce ERRP_AUTO_PROPAGATE
a037a5c virtio-rng: introduce ERRP_AUTO_PROPAGATE
dcf1769 virtio-serial: introduce ERRP_AUTO_PROPAGATE
77d26d1 virtio-input: introduce ERRP_AUTO_PROPAGATE
7f62cb1 virtio-ccw: introduce ERRP_AUTO_PROPAGATE
2bdd860 virtio-blk: introduce ERRP_AUTO_PROPAGATE
026260e virtio-9p: introduce ERRP_AUTO_PROPAGATE
191c845 virtio: introduce ERRP_AUTO_PROPAGATE
24510de vhost: introduce ERRP_AUTO_PROPAGATE
e8a1779 vfio-ccw: introduce ERRP_AUTO_PROPAGATE
00baaa3 VFIO: introduce ERRP_AUTO_PROPAGATE
361c201 USB (serial adapter): introduce ERRP_AUTO_PROPAGATE
0f70e97 USB: introduce ERRP_AUTO_PROPAGATE
9548378 SD (Secure Card): introduce ERRP_AUTO_PROPAGATE
90b472d SCSI: introduce ERRP_AUTO_PROPAGATE
312220a pflash: introduce ERRP_AUTO_PROPAGATE
47a7bb5 Network devices: introduce ERRP_AUTO_PROPAGATE
bf2e1ef ACPI/SMBIOS: introduce ERRP_AUTO_PROPAGATE
98f6d04 PCI: introduce ERRP_AUTO_PROPAGATE
e3e14fe IPack: introduce ERRP_AUTO_PROPAGATE
9ef097c Floppy: introduce ERRP_AUTO_PROPAGATE
4ffbc39 IDE: introduce ERRP_AUTO_PROPAGATE
a91553f X86 Machines: introduce ERRP_AUTO_PROPAGATE
a83188d S390 Machines: introduce ERRP_AUTO_PROPAGATE
8cbe8c9 SPARC Machines: introduce ERRP_AUTO_PROPAGATE

Re: [PATCH v12 06/11] numa: Extend CLI to provide memory latency and bandwidth information

[Tao Xu]

On 10/11/2019 9:56 PM, Igor Mammedov wrote:

On Wed, 9 Oct 2019 14:39:46 +0800
Tao Xu  wrote:


On 10/2/2019 11:16 PM, Igor Mammedov wrote:

On Fri, 20 Sep 2019 15:43:44 +0800
Tao Xu  wrote:
   

[...]

+struct HMAT_LB_Info {
+/* Indicates it's memory or the specified level memory side cache. */
+uint8_t hierarchy;
+
+/* Present the type of data, access/read/write latency or bandwidth. */
+uint8_t data_type;
+
+/* Array to store the latencies */

specify units it's stored in
   

+uint64_t*latency;
+
+/* Array to store the bandwidthes */

ditto
   

+uint64_t*bandwidth;

btw:

what was the reason for picking uint64_t for storing above values?

it seems in this patch you dumb down bandwidth to MB/s above but
store latency as is.


Because I want to store the bandwidth or latency value (minimum unit)
that user input. In HMAT, the minimum unit of bandwidth is MB/s, but in
QAPI, the minimum unit of size is Byte. So I convert size into MB/s and
time unit is "ps", need not convert.

Just be consistent and store (user input) raw values for both fields
(i.e. B/s PS/s) and post-process them later to uint16_t.


and then in 9/11 build_hmat_lb you divide that on 'base' units,
where are guaranties that value stored here will fit into 2 bytes
used in HMAT to store it in the table?
   

For HMAT spec, for a matrix of bandwidth or latency, there is only one
base (in order to save ACPI tables space). We need to extract base for a
matrix, but user input bandwidth or latency line by line. So after all
data input, we can extract the base (as in 9/11).

There is another benefit. If user input different but similar units,
such as "10ns" and "100ps", we can also store them. Only If user input
big gap units, such as "1ps" and "1000ms". we can't store them and raise
error.

No disagreement here,

but I suggest to move verification and base calculation from 09/11
into a separate patch (right after this one) and doing it at
numa_complete_configuration() time.
To store calculated base you can add a common_base field to
sub-table structure (HMAT_LB_Info) and use it when building ACPI
table without extra calculations.



OK, Thank you for your suggestion.

if this structure should store values in terms on HMAT table it should
probably use uint16_t and check that user provided value won't overflow
at the time of CLI parsing.
   

Re: [PATCH 2/2] target/i386: Add support for put/get PEBS registers

[Eduardo Habkost]

On Thu, Aug 29, 2019 at 01:22:55PM +0800, Luwei Kang wrote:
> This patch add a new feature words for IA32_PERF_CAPABILITIES (RO)
> register that serve to expose PEBS output Intel PT feature.
> The registers relate with PEBS need to be set/get when PEBS output
> Intel PT is supported in guest.
> 
> Signed-off-by: Luwei Kang 

Sorry for taking so long to take a look at the series.

What's the status of the kernel KVM patches for this?

> ---
>  target/i386/cpu.c | 20 
>  target/i386/cpu.h |  4 
>  target/i386/kvm.c | 36 
>  3 files changed, 60 insertions(+)
> 
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index 9e0bac3..7fe34c0 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -1244,6 +1244,26 @@ static FeatureWordInfo 
> feature_word_info[FEATURE_WORDS] = {
>  },
>  },
>  },
> +[FEAT_PERF_CAPABILITIES] = {
> +.type = MSR_FEATURE_WORD,
> +.feat_names = {
> +NULL, NULL, NULL, NULL,
> +NULL, NULL, NULL, NULL,
> +NULL, NULL, NULL, NULL,
> +NULL, NULL, NULL, NULL,
> +"pebs-output-pt", NULL, NULL, NULL,
> +NULL, NULL, NULL, NULL,
> +NULL, NULL, NULL, NULL,
> +NULL, NULL, NULL, NULL,
> +},
> +.msr = {
> +.index = MSR_IA32_PERF_CAPABILITIES,
> +.cpuid_dep = {
> +FEAT_1_ECX,
> +CPUID_EXT_PDCM,
> +},
> +},
> +},
>  };
>  
>  typedef struct X86RegisterInfo32 {
> diff --git a/target/i386/cpu.h b/target/i386/cpu.h
> index d7cec36..0904004 100644
> --- a/target/i386/cpu.h
> +++ b/target/i386/cpu.h
> @@ -347,6 +347,7 @@ typedef enum X86Seg {
>  #define MSR_IA32_PRED_CMD   0x49
>  #define MSR_IA32_CORE_CAPABILITY0xcf
>  #define MSR_IA32_ARCH_CAPABILITIES  0x10a
> +#define MSR_IA32_PERF_CAPABILITIES  0x345
>  #define MSR_IA32_TSCDEADLINE0x6e0
>  
>  #define FEATURE_CONTROL_LOCKED(1<<0)
> @@ -503,6 +504,7 @@ typedef enum FeatureWord {
>  FEAT_XSAVE_COMP_HI, /* CPUID[EAX=0xd,ECX=0].EDX */
>  FEAT_ARCH_CAPABILITIES,
>  FEAT_CORE_CAPABILITY,
> +FEAT_PERF_CAPABILITIES,
>  FEATURE_WORDS,
>  } FeatureWord;
>  
> @@ -754,6 +756,8 @@ typedef uint32_t FeatureWordArray[FEATURE_WORDS];
>  
>  #define MSR_CORE_CAP_SPLIT_LOCK_DETECT  (1U << 5)
>  
> +#define MSR_PERF_CAP_PEBS_VIA_PT(1ULL << 16)
> +
>  /* Supported Hyper-V Enlightenments */
>  #define HYPERV_FEAT_RELAXED 0
>  #define HYPERV_FEAT_VAPIC   1
> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
> index 8023c67..c0dcc13 100644
> --- a/target/i386/kvm.c
> +++ b/target/i386/kvm.c
> @@ -2651,6 +2651,20 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
>  kvm_msr_entry_add(cpu, MSR_IA32_RTIT_ADDR0_A + i,
>  env->msr_rtit_addrs[i]);
>  }
> +
> +if (env->features[FEAT_PERF_CAPABILITIES] &
> +MSR_PERF_CAP_PEBS_VIA_PT) {
> +kvm_msr_entry_add(cpu, MSR_IA32_PEBS_ENABLE,
> +env->msr_pebs_enable);
> +for (i = 0; i < num_architectural_pmu_fixed_counters; i++) {
> +kvm_msr_entry_add(cpu, MSR_RELOAD_FIXED_CTR0 + i,
> +env->msr_reload_fixed_ctr[i]);
> +}
> +for (i = 0; i < num_architectural_pmu_gp_counters; i++) {
> +kvm_msr_entry_add(cpu, MSR_RELOAD_PMC0 + i,
> +env->msr_reload_pmc[i]);
> +}
> +}
>  }
>  
>  /* Note: MSR_IA32_FEATURE_CONTROL is written separately, see
> @@ -2989,6 +3003,16 @@ static int kvm_get_msrs(X86CPU *cpu)
>  for (i = 0; i < addr_num; i++) {
>  kvm_msr_entry_add(cpu, MSR_IA32_RTIT_ADDR0_A + i, 0);
>  }
> +
> +if (env->features[FEAT_PERF_CAPABILITIES] & 
> MSR_PERF_CAP_PEBS_VIA_PT) {
> +kvm_msr_entry_add(cpu, MSR_IA32_PEBS_ENABLE, 0);
> +for (i = 0; i < num_architectural_pmu_fixed_counters; i++) {
> +kvm_msr_entry_add(cpu, MSR_RELOAD_FIXED_CTR0 + i, 0);
> +}
> +for (i = 0; i < num_architectural_pmu_gp_counters; i++) {
> +kvm_msr_entry_add(cpu, MSR_RELOAD_PMC0 + i, 0);
> +}
> +}
>  }
>  
>  ret = kvm_vcpu_ioctl(CPU(cpu), KVM_GET_MSRS, cpu->kvm_msr_buf);
> @@ -3268,6 +3292,18 @@ static int kvm_get_msrs(X86CPU *cpu)
>  case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B:
>  env->msr_rtit_addrs[index - MSR_IA32_RTIT_ADDR0_A] = 
> msrs[i].data;
>  break;
> +case MSR_IA32_PEBS_ENABLE:
> +env->msr_pebs_enable = msrs[i].data;
> +break;
> +case MSR_RELOAD_FIXED_CTR0 ...
> +

[PATCH 2/2] migration/compress: disable compress if failed to setup

[Wei Yang]

In current logic, if compress_threads_save_setup() returns -1 the whole
migration would fail, while we could handle it gracefully by disable
compress.

Signed-off-by: Wei Yang 
---
 migration/migration.c |  9 +
 migration/migration.h |  1 +
 migration/ram.c   | 15 ---
 3 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 5f7e4d15e9..02b95f4223 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -2093,6 +2093,15 @@ bool migrate_use_compression(void)
 return s->enabled_capabilities[MIGRATION_CAPABILITY_COMPRESS];
 }
 
+void migrate_disable_compression(void)
+{
+MigrationState *s;
+
+s = migrate_get_current();
+
+s->enabled_capabilities[MIGRATION_CAPABILITY_COMPRESS] = false;
+}
+
 int migrate_compress_level(void)
 {
 MigrationState *s;
diff --git a/migration/migration.h b/migration/migration.h
index 4f2fe193dc..51368d3a6e 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -309,6 +309,7 @@ bool migrate_use_return_path(void);
 uint64_t ram_get_total_transferred_pages(void);
 
 bool migrate_use_compression(void);
+void migrate_disable_compression(void);
 int migrate_compress_level(void);
 int migrate_compress_threads(void);
 int migrate_compress_wait_thread(void);
diff --git a/migration/ram.c b/migration/ram.c
index 96c9b16402..39279161a8 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -533,12 +533,12 @@ static void compress_threads_save_cleanup(void)
 comp_param = NULL;
 }
 
-static int compress_threads_save_setup(void)
+static void compress_threads_save_setup(void)
 {
 int i, thread_count;
 
 if (!migrate_use_compression()) {
-return 0;
+return;
 }
 thread_count = migrate_compress_threads();
 compress_threads = g_new0(QemuThread, thread_count);
@@ -569,11 +569,14 @@ static int compress_threads_save_setup(void)
do_data_compress, comp_param + i,
QEMU_THREAD_JOINABLE);
 }
-return 0;
+return;
 
 exit:
 compress_threads_save_cleanup();
-return -1;
+migrate_disable_compression();
+error_report("%s: failed to setup compress threads, compress disabled",
+ __func__);
+return;
 }
 
 /* Multiple fd's */
@@ -3338,9 +3341,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
 RAMState **rsp = opaque;
 RAMBlock *block;
 
-if (compress_threads_save_setup()) {
-return -1;
-}
+compress_threads_save_setup();
 
 /* migration has already setup the bitmap, reuse it. */
 if (!migration_in_colo_state()) {
-- 
2.17.1

[PATCH 0/2] migration/compress: refine the compress case

[Wei Yang]

Two patches related to compress:

1. simplify the check since compress QEMUFile is not writable
2. handle compress setup failure gracefully

Wei Yang (2):
  migration/compress: compress QEMUFile is not writable
  migration/compress: disable compress if failed to setup

 migration/migration.c |  9 +
 migration/migration.h |  1 +
 migration/qemu-file.c | 16 +++-
 migration/ram.c   | 15 ---
 4 files changed, 21 insertions(+), 20 deletions(-)

-- 
2.17.1

[PATCH 1/2] migration/compress: compress QEMUFile is not writable

[Wei Yang]

We open a file with empty_ops for compress QEMUFile, which means this is
not writable.

Signed-off-by: Wei Yang 
---
 migration/qemu-file.c | 16 +++-
 1 file changed, 3 insertions(+), 13 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 26fb25ddc1..f3d99a96ec 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -744,11 +744,8 @@ static int qemu_compress_data(z_stream *stream, uint8_t 
*dest, size_t dest_len,
 /* Compress size bytes of data start at p and store the compressed
  * data to the buffer of f.
  *
- * When f is not writable, return -1 if f has no space to save the
- * compressed data.
- * When f is wirtable and it has no space to save the compressed data,
- * do fflush first, if f still has no space to save the compressed
- * data, return -1.
+ * Since the file is dummy file with empty_ops, return -1 if f has no space to
+ * save the compressed data.
  */
 ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
   const uint8_t *p, size_t size)
@@ -756,14 +753,7 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream 
*stream,
 ssize_t blen = IO_BUF_SIZE - f->buf_index - sizeof(int32_t);
 
 if (blen < compressBound(size)) {
-if (!qemu_file_is_writable(f)) {
-return -1;
-}
-qemu_fflush(f);
-blen = IO_BUF_SIZE - sizeof(int32_t);
-if (blen < compressBound(size)) {
-return -1;
-}
+return -1;
 }
 
 blen = qemu_compress_data(stream, f->buf + f->buf_index + sizeof(int32_t),
-- 
2.17.1

Re: [PATCH] target/riscv: PMP violation due to wrong size parameter

[Jonathan Behrens]

How do you know that the access won't straddle a page boundary? Is there a
guarantee somewhere that size=0 means that the access is naturally aligned?

Jonathan


On Fri, Oct 11, 2019 at 7:14 PM Dayeol Lee  wrote:

> riscv_cpu_tlb_fill() uses the `size` parameter to check PMP violation
> using pmp_hart_has_privs().
> However, if the size is unknown (=0), the ending address will be
> `addr - 1` as it is `addr + size - 1` in `pmp_hart_has_privs()`.
> This always causes a false PMP violation on the starting address of the
> range, as `addr - 1` is not in the range.
>
> In order to fix, we just assume that all bytes from addr to the end of
> the page will be accessed if the size is unknown.
>
> Signed-off-by: Dayeol Lee 
> Reviewed-by: Richard Henderson 
> ---
>  target/riscv/cpu_helper.c | 13 -
>  1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/target/riscv/cpu_helper.c b/target/riscv/cpu_helper.c
> index e32b6126af..7d9a22b601 100644
> --- a/target/riscv/cpu_helper.c
> +++ b/target/riscv/cpu_helper.c
> @@ -441,6 +441,7 @@ bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address,
> int size,
>  CPURISCVState *env = >env;
>  hwaddr pa = 0;
>  int prot;
> +int pmp_size = 0;
>  bool pmp_violation = false;
>  int ret = TRANSLATE_FAIL;
>  int mode = mmu_idx;
> @@ -460,9 +461,19 @@ bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address,
> int size,
>"%s address=%" VADDR_PRIx " ret %d physical "
> TARGET_FMT_plx
>" prot %d\n", __func__, address, ret, pa, prot);
>
> +/*
> + * if size is unknown (0), assume that all bytes
> + * from addr to the end of the page will be accessed.
> + */
> +if (size == 0) {
> +pmp_size = -(address | TARGET_PAGE_MASK);
> +} else {
> +pmp_size = size;
> +}
> +
>  if (riscv_feature(env, RISCV_FEATURE_PMP) &&
>  (ret == TRANSLATE_SUCCESS) &&
> -!pmp_hart_has_privs(env, pa, size, 1 << access_type, mode)) {
> +!pmp_hart_has_privs(env, pa, pmp_size, 1 << access_type, mode)) {
>  ret = TRANSLATE_PMP_FAIL;
>  }
>  if (ret == TRANSLATE_PMP_FAIL) {
> --
> 2.20.1
>
>
>

Re: [RFC v5 000/126] error: auto propagated local_err

[no-reply]

Patchew URL: 
https://patchew.org/QEMU/20191011160552.22907-1-vsement...@virtuozzo.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  TESTiotest-qcow2: 013
  TESTcheck-unit: tests/test-block-iothread
  TESTcheck-unit: tests/test-image-locking
warning: Failed to get shared "consistent read" lock
Is another process using the image [/tmp/qtest.O66l3t]?
warning: Failed to get shared "consistent read" lock
Is another process using the image [/tmp/qtest.G0M9V8]?
  TESTcheck-unit: tests/test-x86-cpuid
  TESTcheck-unit: tests/test-xbzrle
---
  TESTiotest-qcow2: 267
Failures: 054
Failed 1 of 108 iotests
make: *** [check-tests/check-block.sh] Error 1
make: *** Waiting for unfinished jobs
  TESTcheck-qtest-aarch64: tests/qos-test
Traceback (most recent call last):
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', 
'--label', 'com.qemu.instance.uuid=2c55fb61a63a409382f2b3a99fca93d9', '-u', 
'1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', 
'-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 
'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', 
'/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', 
'/var/tmp/patchew-tester-tmp-usvi8fob/src/docker-src.2019-10-11-22.00.10.1933:/var/tmp/qemu:z,ro',
 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit 
status 2.
filter=--filter=label=com.qemu.instance.uuid=2c55fb61a63a409382f2b3a99fca93d9
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-usvi8fob/src'
make: *** [docker-run-test-quick@centos7] Error 2

real10m33.714s
user0m8.360s


The full log is available at
http://patchew.org/logs/20191011160552.22907-1-vsement...@virtuozzo.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

Re: [PATCH v2] target/i386: Add Snowridge-v2 (noMPX) CPU model

[Eduardo Habkost]

On Sat, Oct 12, 2019 at 09:31:25AM +0800, Xiaoyao Li wrote:
> On 10/12/2019 9:21 AM, Eduardo Habkost wrote:
> > On Sat, Oct 12, 2019 at 09:15:56AM +0800, Xiaoyao Li wrote:
> > > On 10/12/2019 2:21 AM, Eduardo Habkost wrote:
> > > > On Fri, Oct 11, 2019 at 10:53:49PM +0800, Xiaoyao Li wrote:
> > > > > Add new version of Snowridge CPU model that removes MPX feature.
> > > > > 
> > > > > MPX support is being phased out by Intel. GCC has dropped it, Linux 
> > > > > kernel
> > > > > and kvm are also going to do that in the future.
> > > > > 
> > > > > Signed-off-by: Xiaoyao Li 
> > > > > ---
> > > > > Changes in v2:
> > > > >   - Use CPU model versioning mechanism instead of machine-type 
> > > > > compat
> > > > > ---
> > > > >target/i386/cpu.c | 13 +
> > > > >1 file changed, 13 insertions(+)
> > > > > 
> > > > > diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> > > > > index 44f1bbdcac76..27b0a17b46a8 100644
> > > > > --- a/target/i386/cpu.c
> > > > > +++ b/target/i386/cpu.c
> > > > > @@ -2793,6 +2793,19 @@ static X86CPUDefinition builtin_x86_defs[] = {
> > > > >CPUID_6_EAX_ARAT,
> > > > >.xlevel = 0x8008,
> > > > >.model_id = "Intel Atom Processor (SnowRidge)",
> > > > > +.versions = (X86CPUVersionDefinition[]) {
> > > > > +{ .version = 1 },
> > > > > +{
> > > > > +.version = 2,
> > > > > +.alias = "Snowridge-noMPX",
> > > > 
> > > > The intention is to stop creating new funny names for CPU model
> > > > variations, now, and stick to -v1, -v2, -v3, etc.
> > > > 
> > > > The .alias field is optional, and was added only for
> > > > compatibility with the existing -noTSX and -IBRS CPU models.
> > > 
> > > Got it.
> > > 
> > > > > +.props = (PropValue[]) {
> > > > > +{ "mpx", "off" },
> > > > > +{ "model-id", "Intel Atom Processor (Snowridge, 
> > > > > no MPX)" },
> > > > 
> > > > Do you think it's important to report a different model-id?
> > > > I would keep it the same and only add mpx=off.
> > > 
> > > I just want to let user know easily the differences between Snowridge-v1 
> > > and
> > > Snowridge-v2. Unfortunately, it seems ugly.
> > > 
> > > When testing with Cascadelake-Server, it puzzles every time that which one
> > > should I choose between Cascadelake-Server-v1 and Cascadelake-Server-v2.
> > >  From the output of "-cpu ?", I don't know the differences between them.
> > > Everytime I have to go to the source code to see the difference.
> > > 
> > > Maybe there is a way to see/report the differences between different
> > > versions of the same CPU model that I just don't know?
> > 
> > Good point.  I forgot that model-id is also the model description
> > in "-cpu ?".
> > 
> > Well, it doesn't hurt to have a different model-id in v2 that's
> > more informative.  Feel free to keep the model-id line in v3.
> 
> OK. I will send out the v3 patch keeping the model-id while removing the
> alias.
> 
> BTW, do you have better idea to tell the differences among versions of the
> same CPU model?

We could add a new field for a human-readable description of the
CPU model version, and print that field on "-cpu help" if set.

We could also try to generate a description automatically (e.g.
automatically describe SnowRidge-v2 as "SnowRidge-v1 plus mpx"
based on .props).  I'm not sure if it would be worth the effort,
though.

-- 
Eduardo

Re: [PATCH v2] target/i386: Add Snowridge-v2 (noMPX) CPU model

[Xiaoyao Li]

On 10/12/2019 9:21 AM, Eduardo Habkost wrote:

On Sat, Oct 12, 2019 at 09:15:56AM +0800, Xiaoyao Li wrote:

On 10/12/2019 2:21 AM, Eduardo Habkost wrote:

On Fri, Oct 11, 2019 at 10:53:49PM +0800, Xiaoyao Li wrote:

Add new version of Snowridge CPU model that removes MPX feature.

MPX support is being phased out by Intel. GCC has dropped it, Linux kernel
and kvm are also going to do that in the future.

Signed-off-by: Xiaoyao Li 
---
Changes in v2:
  - Use CPU model versioning mechanism instead of machine-type compat
---
   target/i386/cpu.c | 13 +
   1 file changed, 13 insertions(+)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 44f1bbdcac76..27b0a17b46a8 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -2793,6 +2793,19 @@ static X86CPUDefinition builtin_x86_defs[] = {
   CPUID_6_EAX_ARAT,
   .xlevel = 0x8008,
   .model_id = "Intel Atom Processor (SnowRidge)",
+.versions = (X86CPUVersionDefinition[]) {
+{ .version = 1 },
+{
+.version = 2,
+.alias = "Snowridge-noMPX",


The intention is to stop creating new funny names for CPU model
variations, now, and stick to -v1, -v2, -v3, etc.

The .alias field is optional, and was added only for
compatibility with the existing -noTSX and -IBRS CPU models.


Got it.


+.props = (PropValue[]) {
+{ "mpx", "off" },
+{ "model-id", "Intel Atom Processor (Snowridge, no MPX)" },


Do you think it's important to report a different model-id?
I would keep it the same and only add mpx=off.


I just want to let user know easily the differences between Snowridge-v1 and
Snowridge-v2. Unfortunately, it seems ugly.

When testing with Cascadelake-Server, it puzzles every time that which one
should I choose between Cascadelake-Server-v1 and Cascadelake-Server-v2.
 From the output of "-cpu ?", I don't know the differences between them.
Everytime I have to go to the source code to see the difference.

Maybe there is a way to see/report the differences between different
versions of the same CPU model that I just don't know?


Good point.  I forgot that model-id is also the model description
in "-cpu ?".

Well, it doesn't hurt to have a different model-id in v2 that's
more informative.  Feel free to keep the model-id line in v3.


OK. I will send out the v3 patch keeping the model-id while removing the 
alias.


BTW, do you have better idea to tell the differences among versions of 
the same CPU model?

Re: [PATCH v2] target/i386: Add Snowridge-v2 (noMPX) CPU model

[Eduardo Habkost]

On Sat, Oct 12, 2019 at 09:15:56AM +0800, Xiaoyao Li wrote:
> On 10/12/2019 2:21 AM, Eduardo Habkost wrote:
> > On Fri, Oct 11, 2019 at 10:53:49PM +0800, Xiaoyao Li wrote:
> > > Add new version of Snowridge CPU model that removes MPX feature.
> > > 
> > > MPX support is being phased out by Intel. GCC has dropped it, Linux kernel
> > > and kvm are also going to do that in the future.
> > > 
> > > Signed-off-by: Xiaoyao Li 
> > > ---
> > > Changes in v2:
> > >  - Use CPU model versioning mechanism instead of machine-type compat
> > > ---
> > >   target/i386/cpu.c | 13 +
> > >   1 file changed, 13 insertions(+)
> > > 
> > > diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> > > index 44f1bbdcac76..27b0a17b46a8 100644
> > > --- a/target/i386/cpu.c
> > > +++ b/target/i386/cpu.c
> > > @@ -2793,6 +2793,19 @@ static X86CPUDefinition builtin_x86_defs[] = {
> > >   CPUID_6_EAX_ARAT,
> > >   .xlevel = 0x8008,
> > >   .model_id = "Intel Atom Processor (SnowRidge)",
> > > +.versions = (X86CPUVersionDefinition[]) {
> > > +{ .version = 1 },
> > > +{
> > > +.version = 2,
> > > +.alias = "Snowridge-noMPX",
> > 
> > The intention is to stop creating new funny names for CPU model
> > variations, now, and stick to -v1, -v2, -v3, etc.
> > 
> > The .alias field is optional, and was added only for
> > compatibility with the existing -noTSX and -IBRS CPU models.
> 
> Got it.
> 
> > > +.props = (PropValue[]) {
> > > +{ "mpx", "off" },
> > > +{ "model-id", "Intel Atom Processor (Snowridge, no 
> > > MPX)" },
> > 
> > Do you think it's important to report a different model-id?
> > I would keep it the same and only add mpx=off.
> 
> I just want to let user know easily the differences between Snowridge-v1 and
> Snowridge-v2. Unfortunately, it seems ugly.
> 
> When testing with Cascadelake-Server, it puzzles every time that which one
> should I choose between Cascadelake-Server-v1 and Cascadelake-Server-v2.
> From the output of "-cpu ?", I don't know the differences between them.
> Everytime I have to go to the source code to see the difference.
> 
> Maybe there is a way to see/report the differences between different
> versions of the same CPU model that I just don't know?

Good point.  I forgot that model-id is also the model description
in "-cpu ?".

Well, it doesn't hurt to have a different model-id in v2 that's
more informative.  Feel free to keep the model-id line in v3.

Thanks!

-- 
Eduardo

Re: [PATCH v2] target/i386: Add Snowridge-v2 (noMPX) CPU model

[Xiaoyao Li]

On 10/12/2019 2:21 AM, Eduardo Habkost wrote:

On Fri, Oct 11, 2019 at 10:53:49PM +0800, Xiaoyao Li wrote:

Add new version of Snowridge CPU model that removes MPX feature.

MPX support is being phased out by Intel. GCC has dropped it, Linux kernel
and kvm are also going to do that in the future.

Signed-off-by: Xiaoyao Li 
---
Changes in v2:
 - Use CPU model versioning mechanism instead of machine-type compat
---
  target/i386/cpu.c | 13 +
  1 file changed, 13 insertions(+)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 44f1bbdcac76..27b0a17b46a8 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -2793,6 +2793,19 @@ static X86CPUDefinition builtin_x86_defs[] = {
  CPUID_6_EAX_ARAT,
  .xlevel = 0x8008,
  .model_id = "Intel Atom Processor (SnowRidge)",
+.versions = (X86CPUVersionDefinition[]) {
+{ .version = 1 },
+{
+.version = 2,
+.alias = "Snowridge-noMPX",


The intention is to stop creating new funny names for CPU model
variations, now, and stick to -v1, -v2, -v3, etc.

The .alias field is optional, and was added only for
compatibility with the existing -noTSX and -IBRS CPU models.


Got it.


+.props = (PropValue[]) {
+{ "mpx", "off" },
+{ "model-id", "Intel Atom Processor (Snowridge, no MPX)" },


Do you think it's important to report a different model-id?
I would keep it the same and only add mpx=off.


I just want to let user know easily the differences between Snowridge-v1 
and Snowridge-v2. Unfortunately, it seems ugly.


When testing with Cascadelake-Server, it puzzles every time that which 
one should I choose between Cascadelake-Server-v1 and 
Cascadelake-Server-v2. From the output of "-cpu ?", I don't know the 
differences between them. Everytime I have to go to the source code to 
see the difference.


Maybe there is a way to see/report the differences between different 
versions of the same CPU model that I just don't know?



+{ /* end of list */ },
+},
+},
+{ /* end of list */ },
+},
  },
  {
  .name = "KnightsMill",
--
2.19.1

Re: [Bug 1805256] Re: [Qemu-devel] qemu_futex_wait() lockups in ARM64: 2 possible issues

[no-reply]

Patchew URL: https://patchew.org/QEMU/20191011175536.GB25464@xps13.dannf/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Bug 1805256] Re: [Qemu-devel] qemu_futex_wait() lockups in ARM64: 2 
possible issues
Type: series
Message-id: 20191011175536.GB25464@xps13.dannf

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
d3707c3 qemu_futex_wait() lockups in ARM64: 2 possible issues

=== OUTPUT BEGIN ===
WARNING: Block comments use a leading /* on a separate line
#66: FILE: util/async.c:345:
+/* Using atomic_mb_read ensures that e.g. bh->scheduled is written before

ERROR: code indent should never use tabs
#74: FILE: util/async.c:350:
+^Ievent_notifier_set(>notifier);$

ERROR: Missing Signed-off-by: line(s)

total: 2 errors, 1 warnings, 33 lines checked

Commit d3707c31b5da (qemu_futex_wait() lockups in ARM64: 2 possible issues) has 
style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20191011175536.GB25464@xps13.dannf/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

Re: [Bug 1805256] Re: [Qemu-devel] qemu_futex_wait() lockups in ARM64: 2 possible issues

[no-reply]

Patchew URL: https://patchew.org/QEMU/20191011175536.GB25464@xps13.dannf/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing 
commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

In file included from /tmp/qemu-test/src/include/qemu/osdep.h:51,
 from /tmp/qemu-test/src/util/async.c:26:
/tmp/qemu-test/src/util/async.c: In function 'aio_notify':
/tmp/qemu-test/src/include/qemu/compiler.h:86:36: error: static assertion 
failed: "not expecting: sizeof(*>notify_me) > ATOMIC_REG_SIZE"
 #define QEMU_BUILD_BUG_MSG(x, msg) _Static_assert(!(x), msg)
^~
/tmp/qemu-test/src/include/qemu/compiler.h:94:30: note: in expansion of macro 
'QEMU_BUILD_BUG_MSG'
---
/tmp/qemu-test/src/util/async.c:349:9: note: in expansion of macro 
'atomic_mb_read'
 if (atomic_mb_read(>notify_me)) {
 ^~
make: *** [/tmp/qemu-test/src/rules.mak:69: util/async.o] Error 1
make: *** Waiting for unfinished jobs
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 662, in 
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', 
'--label', 'com.qemu.instance.uuid=b90449ea684e4a14ad6b34f4f5dcb0c9', '-u', 
'1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', 
'-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 
'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', 
'/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', 
'/var/tmp/patchew-tester-tmp-0y2eb54d/src/docker-src.2019-10-11-20.12.57.30254:/var/tmp/qemu:z,ro',
 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit 
status 2.
filter=--filter=label=com.qemu.instance.uuid=b90449ea684e4a14ad6b34f4f5dcb0c9
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-0y2eb54d/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real11m27.735s
user0m7.532s


The full log is available at
http://patchew.org/logs/20191011175536.GB25464@xps13.dannf/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

Re: Exploring Sphinx, autodoc, apidoc, and coverage tools for python/qemu

[John Snow]

On 7/24/19 5:06 PM, John Snow wrote:
> Has anyone on this list experimented with these tools?
> 
> I was hoping to use them to document things like the python/machine.py
> and python/qmp.py modules to help demonstrate some of our internal
> tooling API (for test writers, GSoC/Outreachy interns, folks who want to
> script QEMU at a level between writing a CLI driver and using libvirt.)
> 
> What follows below is my process trying to enable this and some of the
> problems I'm still stuck with, summarized below at the end of this more
> exploratory text.
> 
> 
> Enabling autodoc:
> 
> First, it appears as if enabling the "sphinx-autodoc" tool is not
> sufficient for actually generating anything at all when you invoke the
> sphinx-generated "make html" target. It just enables understanding
> certain directives.
> 
> So apparently you need to generate module "stubs" using sphinx-autodoc.
> Sphinx uses the sphinx-autodoc extension to understand how to consume
> the directives in these stubs.
> 
> That strikes me as odd, because these stubs might need to be changed
> frequently as code comes and goes; it seems strange that it isn't
> integrated at the top level. (Do I have the wrong idea on how these
> tools should be used?)
> 
> So you need to run:
>> sphinx-apidoc --separate --module-first -o docs/ python/qemu/
> 
> which generates stubs to docs:
> 
> Creating file docs/qemu.machine.rst.
> Creating file docs/qemu.qmp.rst.
> Creating file docs/qemu.qtest.rst.
> Creating file docs/qemu.rst.
> Creating file docs/modules.rst.
> 
> And then you can edit e.g. the top-level index.rst TOC in docs/index.rst
> to look like this:
> 
> ```
> .. toctree::
>:maxdepth: 2
>:caption: Contents:
> 
>interop/index
>devel/index
>specs/index
>modules
> ```
> 
> And then finally generating the build; manually removing the -W option
> from the Makefile: there are a lot of warnings in here.
> 
>> sphinx-build -n -b html -D version=4.0.92 -D release="4.0.92
> (v4.1.0-rc2-34-g160802eb07-dirty)" -d .doctrees/
> /home/bos/jhuston/src/qemu/docs/ docs/
> 
> Great! that will generate output to docs/index.html which indeed shows
> APIdoc comments generated from our Python files. Good.
> 
> However, where this gets a little strange is if you look at the
> generated stubs. For instance, qemu.machine.rst looks like this:
> 
> ```
> .. automodule:: qemu.machine
> :members:
> :undoc-members:
> :show-inheritance:
> ```
> 
> :undoc-members: says that we want to "document" any members that don't
> have a matching apidoc comment by generating a stub.
> 
> Oops, but the presence of that stub will cause the sphinx coverage tool
> to happily report 100% coverage.
> 
> Further oops, pylint doesn't understand apidoc comments and can't be
> used as the linter in this case, either.
> 
> You can edit the stubs to remove these directives, but these stubs are
> generated -- and it doesn't appear like there's a command line option to
> change this behavior. ...Hmm.
> 

Update: there is indeed a way to change this behavior, through
environment variables.

https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html

SPHINX_APIDOC_OPTIONS
A comma-separated list of option to append to generated automodule
directives. Defaults to members,undoc-members,show-inheritance.

You just have to set it and remove 'undoc-members'.

> And either way, the coverage tool only generates a report and not
> something with an error code that I could use to gate the build. Same
> goes for the general build: if I remove the :undoc-members: parameter,
> there's nothing in the autodoc module that appears to throw warnings
> when it encounters undocumented parameters or members.
> 

This is still a problem, though. Nothing gates on undocumented members
at all.

> That seems disappointing, because it's hard to keep docstrings up to
> date unless they are checked conclusively at build time.
> 
> 
> Conclusions:
> 
> - the autodoc documentation page doesn't seem to document examples of
> how you're expected to write meaningful docstrings for the tool to extract.
> 
> - autodoc fools the coverage tool into reporting 100% coverage.
> 
> - autodoc can be configured to omit non-documented members to allow the
> coverage tool to work, but the configuration is auto-generated and
> defaults to always generating documentation for these entities.
> 

apidoc (the stub generator for autodoc) can indeed be configured to omit
non-documented members through an environment variable now, so this
point is not true.

> - coverage tool doesn't appear like it can be used for gating the build
> natively for missing python docs; it only generates a report.
> 

Still seems true, though there are other tools that can gate on such
things. Specifically we can use pylint and require that docstrings are
present.

Notably, pylint only checks that there is a docstring at all, and
doesn't try to perform any semantic validation of what's in it.

To my knowledge, there are no standalone 

Re: [PATCH v2] iotests/028: Fix for long $TEST_DIRs

[John Snow]

On 10/11/19 8:18 AM, Max Reitz wrote:
> For long test image paths, the order of the "Formatting" line and the
> "(qemu)" prompt after a drive_backup HMP command may be reversed.  In
> fact, the interaction between the prompt and the line may lead to the
> "Formatting" to being greppable at all after "read"-ing it (if the
> prompt injects an IFS character into the "Formatting" string).
> 
> So just wait until we get a prompt.  At that point, the block job must
> have been started, so "info block-jobs" will only return "No active
> jobs" once it is done.
> 
> Reported-by: Thomas Huth 
> Signed-off-by: Max Reitz 
> ---
> v2:
> - Fix another kind of race...
> ---
>  tests/qemu-iotests/028 | 11 ---
>  tests/qemu-iotests/028.out |  1 -
>  2 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/qemu-iotests/028 b/tests/qemu-iotests/028
> index 71301ec6e5..bba1ee59ae 100755
> --- a/tests/qemu-iotests/028
> +++ b/tests/qemu-iotests/028
> @@ -119,9 +119,14 @@ fi
>  # Silence output since it contains the disk image path and QEMU's readline
>  # character echoing makes it very hard to filter the output. Plus, there
>  # is no telling how many times the command will repeat before succeeding.
> -_send_qemu_cmd $h "drive_backup disk ${TEST_IMG}.copy" "(qemu)" >/dev/null
> -_send_qemu_cmd $h "" "Formatting" | _filter_img_create
> -qemu_cmd_repeat=20 _send_qemu_cmd $h "info block-jobs" "No active jobs" 
> >/dev/null
> +# (Note that creating the image results in a "Formatting..." message over
> +# stdout, which is the same channel the monitor uses.  We cannot reliably
> +# wait for it because the monitor output may interact with it in such a
> +# way that _timed_wait_for cannot read it.  However, once the block job is
> +# done, we know that the "Formatting..." message must have appeared
> +# already, so the output is still deterministic.)
> +silent=y _send_qemu_cmd $h "drive_backup disk ${TEST_IMG}.copy" "(qemu)"
> +silent=y qemu_cmd_repeat=20 _send_qemu_cmd $h "info block-jobs" "No active 
> jobs"
>  _send_qemu_cmd $h "info block-jobs" "No active jobs"
>  _send_qemu_cmd $h 'quit' ""
>  
> diff --git a/tests/qemu-iotests/028.out b/tests/qemu-iotests/028.out
> index 7d54aeb003..37aed84436 100644
> --- a/tests/qemu-iotests/028.out
> +++ b/tests/qemu-iotests/028.out
> @@ -468,7 +468,6 @@ No errors were found on the image.
>  
>  block-backup
>  
> -Formatting 'TEST_DIR/t.IMGFMT.copy', fmt=IMGFMT size=4294968832 
> backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
>  (qemu) info block-jobs
>  No active jobs
>  === IO: pattern 195
> 

Reviewed-by: John Snow 

Re: [PATCH v4 17/19] spapr: Remove last pieces of SpaprIrq

[David Gibson]

On Fri, Oct 11, 2019 at 08:13:33AM +0200, Greg Kurz wrote:
> On Fri, 11 Oct 2019 16:07:58 +1100
> David Gibson  wrote:
> 
> > On Thu, Oct 10, 2019 at 10:33:04PM +0200, Greg Kurz wrote:
> > > On Thu, 10 Oct 2019 08:29:58 +0200
> > > Greg Kurz  wrote:
> > > 
> > > > On Thu, 10 Oct 2019 13:02:09 +1100
> > > > David Gibson  wrote:
> > > > 
> > > > > On Wed, Oct 09, 2019 at 07:02:15PM +0200, Greg Kurz wrote:
> > > > > > On Wed,  9 Oct 2019 17:08:16 +1100
> > > > > > David Gibson  wrote:
> > > > > > 
> > > > > > > The only thing remaining in this structure are the flags to allow 
> > > > > > > either
> > > > > > > XICS or XIVE to be present.  These actually make more sense as 
> > > > > > > spapr
> > > > > > > capabilities - that way they can take advantage of the existing
> > > > > > > infrastructure to sanity check capability states across migration 
> > > > > > > and so
> > > > > > > forth.
> > > > > > > 
> > > > > > 
> > > > > > The user can now choose the interrupt controller mode either through
> > > > > > ic-mode or through cap-xics/cap-xive. I guess it doesn't break 
> > > > > > anything
> > > > > > to expose another API to do the same thing but it raises some 
> > > > > > questions.
> > > > > > 
> > > > > > We should at least document somewhere that ic-mode is an alias to 
> > > > > > these
> > > > > > caps, and maybe state which is the preferred method (I personally 
> > > > > > vote
> > > > > > for the caps).
> > > > > > 
> > > > > > Also, we must keep ic-mode for the moment to stay compatible with 
> > > > > > the
> > > > > > existing pseries-4.0 and pseries-4.1 machine types, but will we
> > > > > > keep ic-mode forever ? If no, maybe start by not allowing it for
> > > > > > pseries-4.2 ?
> > > > > 
> > > > > I'm actually inclined to keep it for now, maybe even leave it as the
> > > > > suggested way to configure this.  The caps are nice from an internal
> > > > > organization point of view, but ic-mode is arguably a more user
> > > > > friendly way of configuring it.  The conversion of one to the other is
> > > > > straightforward, isolated ans small, so I'm not especially bothered by
> > > > > keeping it around.
> > > > > 
> > > > 
> > > > Fair enough.
> > > > 
> > > > Reviewed-by: Greg Kurz 
> > > > 
> > > 
> > > But unfortunately this still requires care :-\
> > > 
> > > qemu-system-ppc64: cap-xive higher level (1) in incoming stream than on 
> > > destination (0)
> > > qemu-system-ppc64: error while loading state for instance 0x0 of device 
> > > 'spapr'
> > > qemu-system-ppc64: load of migration failed: Invalid argument
> > > 
> > > or
> > > 
> > > qemu-system-ppc64: cap-xics higher level (1) in incoming stream than on 
> > > destination (0)
> > > qemu-system-ppc64: error while loading state for instance 0x0 of device 
> > > 'spapr'
> > > qemu-system-ppc64: load of migration failed: Invalid argument
> > > 
> > > when migrating from QEMU 4.1 with ic-mode=xics and ic-mode=xive 
> > > respectively.
> > > 
> > > This happens because the existing pseries-4.1 machine type doesn't send 
> > > the
> > > new caps and the logic in spapr_caps_post_migration() wrongly assumes that
> > > the source has both caps set:
> > > 
> > > srccaps = default_caps_with_cpu(spapr, MACHINE(spapr)->cpu_type);
> > > for (i = 0; i < SPAPR_CAP_NUM; i++) {
> > > /* If not default value then assume came in with the migration */
> > > if (spapr->mig.caps[i] != spapr->def.caps[i]) {
> > > 
> > > spapr->mig.caps[SPAPR_CAP_XICS] = 0
> > > spapr->mig.caps[SPAPR_CAP_XIVE] = 0
> > > 
> > > srccaps.caps[i] = spapr->mig.caps[i];
> > > 
> > > srcaps.caps[SPAPR_CAP_XICS] = 1
> > > srcaps.caps[SPAPR_CAP_XIVE] = 1
> > > 
> > > }
> > > }
> > > 
> > > and breaks
> > > 
> > > for (i = 0; i < SPAPR_CAP_NUM; i++) {
> > > SpaprCapabilityInfo *info = _table[i];
> > > 
> > > if (srccaps.caps[i] > dstcaps.caps[i]) {
> > > 
> > > srcaps.caps[SPAPR_CAP_XICS] = 0 when ic-mode=xive
> > > srcaps.caps[SPAPR_CAP_XIVE] = 0 when ic-mode=xics
> > > 
> > > error_report("cap-%s higher level (%d) in incoming stream 
> > > than on destination (%d)",
> > >  info->name, srccaps.caps[i], dstcaps.caps[i]);
> > > ok = false;
> > > }
> > 
> > Ah.. right.  I thought there would be problems with backwards
> > migration, but I didn't think of this problem even with forward
> > migration.
> > 
> > > Maybe we shouldn't check capabilities that we know the source
> > > isn't supposed to send, eg. by having a smc->max_cap ?
> > 
> > Uh.. I'm not really sure what exactly you're suggesting here.
> > 
> 
> I'm suggesting to have a per-machine version smc->max_cap that
> contains the highest supported cap index, to be used instead of
> SPAPR_CAP_NUM in this functions, ie.
> 
> for (i = 0; i <= smc->max_cap; i++) {
> ...
> }
> 
> where we would have
> 
> smc->max_cap = SPAPR_CAP_CCF_ASSIST for pseries-4.1
> 
> and
> 
> smc->max_cap = SPAPR_CAP_XIVE for 

Re: [PATCH v5 0/5] iotests: use python logging

[John Snow]

On 10/4/19 11:39 AM, Max Reitz wrote:
> On 18.09.19 01:45, John Snow wrote:
>> This series uses python logging to enable output conditionally on
>> iotests.log(). We unify an initialization call (which also enables
>> debugging output for those tests with -d) and then make the switch
>> inside of iotests.
>>
>> It will help alleviate the need to create logged/unlogged versions
>> of all the various helpers we have made.
>>
>> V5:
>>  - Rebased again
>>  - Allow Python tests to run on any platform
>>
>> V4:
>>  - Rebased on top of kwolf/block at the behest of mreitz
>>
>> V3:
>>  - Rebased for 4.1+; now based on main branch.
>>
>> V2:
>>  - Added all of the other python tests I missed to use script_initialize
>>  - Refactored the common setup as per Ehabkost's suggestion
>>  - Added protocol arguments to common initialization,
>>but this isn't strictly required.
> 
> I’m OK to take the series as-is (it doesn’t affect any auto tests, so we
> can decide what to do about non-Linux platforms in make check at a later
> point), but there seems to be something you wanted to fix up in patch 5.
> 
> (And there’s also Kevin’s pending pull request that changes a bit of
> iotests.py.)
> 
> Max
> 

Just caught up with the discussion.

It looks like Thomas took my 1/5; so I'll respin on top of his "[PATCH
0/5] Enable more iotests during "make check-block" series to catch those
improvements as they stand.

--js

Re: [PULL 01/19] util/hbitmap: strict hbitmap_reset

[John Snow]

On 10/11/19 5:48 PM, Eric Blake wrote:
> On 10/11/19 4:25 PM, John Snow wrote:
>> From: Vladimir Sementsov-Ogievskiy 
>>
>> hbitmap_reset has an unobvious property: it rounds requested region up.
>> It may provoke bugs, like in recently fixed write-blocking mode of
>> mirror: user calls reset on unaligned region, not keeping in mind that
>> there are possible unrelated dirty bytes, covered by rounded-up region
>> and information of this unrelated "dirtiness" will be lost.
>>
>> Make hbitmap_reset strict: assert that arguments are aligned, allowing
>> only one exception when @start + @count == hb->orig_size. It's needed
>> to comfort users of hbitmap_next_dirty_area, which cares about
>> hb->orig_size.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy 
>> Reviewed-by: Max Reitz 
>> Message-Id: <20190806152611.280389-1-vsement...@virtuozzo.com>
>> [Maintainer edit: Max's suggestions from on-list. --js]
>> Signed-off-by: John Snow 
>> ---
>>   include/qemu/hbitmap.h | 5 +
>>   tests/test-hbitmap.c   | 2 +-
>>   util/hbitmap.c | 4 
>>   3 files changed, 10 insertions(+), 1 deletion(-)
>>
> 
>> +++ b/util/hbitmap.c
>> @@ -476,6 +476,10 @@ void hbitmap_reset(HBitmap *hb, uint64_t start,
>> uint64_t count)
>>   /* Compute range in the last layer.  */
>>   uint64_t first;
>>   uint64_t last = start + count - 1;
>> +    uint64_t gran = 1ULL << hb->granularity;
>> +
>> +    assert(!(start & (gran - 1)));
>> +    assert(!(count & (gran - 1)) || (start + count == hb->orig_size));
> 
> I know I'm replying a bit late (since this is now a pull request), but
> would it be worth using the dedicated macro:
> 
> assert(QEMU_IS_ALIGNED(start, gran));
> assert(QEMU_IS_ALIGNED(count, gran) || start + count == hb->orig_size);
> 
> instead of open-coding it?  (I would also drop the extra () around the
> right half of ||). If we want it, that would now be a followup patch.
> 

If the PR doesn't make it for some reason, I can amend a cleanup patch
for the next PR.

--js

Re: [PATCH v2 1/1] target/riscv/pmp: Fix bug preventing

[Dayeol Lee]

Hi, Alistair,

Thank you for reminding me.
I already had the local patch, so I re-submitted the patch.
Please let me know if that's fair enough (or you have any other comments)

Thanks,

Dayeol

On Fri, Oct 11, 2019 at 3:24 PM Alistair Francis 
wrote:

> On Sun, Oct 6, 2019 at 1:32 AM Chris Williams  wrote:
> >
> > Hello. I hope you don't mind me resubmitting this patch. Please let me
> know if
>
> Not at all!
>
> Thanks for the patch.
>
> In future if people don't respond you can just keep pinging your patch.
>
> > I've formatted it incorrectly or if it needs more explanation. My
> previous
> > attempt probably wasn't formatted quite right. This is my first time
> > contributing to Qemu, so I'm keen to get it right - thanks.
>
> This whole paragraph should not be in the commit. It can go below the
> line though.
>
> Also please use `git format-patch` to format the patch and then `git
> send-email` to send the patch. There is a whole heap of detail here:
> https://wiki.qemu.org/Contribute/SubmitAPatch
>
> >
> > This fixes an issue that prevents a RISC-V CPU from executing
> instructions
> > immediately from the base address of a PMP TOR region.
> >
> > When jumping to an instruction in a PMP TOR region, pmp_hart_has_privs()
> is
> > called to validate the access. If this instruction is the very first
> word of a
> > PMP TOR region, at address 0 relative to the start address of the
> region, then
> > the access will fail. This is because pmp_hart_has_privs() is called
> with size
> > 0 to perform this validation, causing this check...
> >
> > e = pmp_is_in_range(env, i, addr + size - 1);
> >
> > ... to fail, as (addr + size - 1) falls below the base address of the PMP
> > region. Really, the access should succeed. For example, if I have a
> region
> > spanning 0x80d96000 to 0x88d95fff and the CPU jumps to 0x80d96000, then:
> >
> > s = 0x80d96000
> > e = 0x80d95fff
> >
> > And the validation fails. The size check proposed below catches these
> zero-size
> > instruction fetch access probes. The word alignment in pmpaddr{0-15} and
> > earlier instruction alignment checks should prevent the execution of
> > instructions over the upper boundary of the PMP region, though I'm happy
> to give
> > this more attention if this is a concern.
>
> This seems like a similar issue to this patch as well:
>
> https://lore.kernel.org/qemu-devel/20191007052813.25814-1-day...@berkeley.edu/
>
> From that discussion:
>
> "In general, size 0 means "unknown size".  In this case, the one tlb
> lookup is
> going to be used by lots of instructions -- everything that fits on the
> page."
>
> Richard's last comment seems like a better fix:
>
> "You certainly could do
>
> if (size == 0) {
> size = -(addr | TARGET_PAGE_MASK);
> }
>
> to assume that all bytes from addr to the end of the page are accessed.
> That
> would avoid changing too much of the rest of the logic.
>
> That said, this code will continue to not work for mis-aligned boundaries."
>
> So I don't think this is the correct solution. I'm not sure if Dayeol
> is planning on sending a follow up version. If not feel free to send
> it.
>
> >
> > Signed-off-by: Chris Williams  diodes...@tuta.io>>
>
> It looks like this is a HTML patch, also ensure all patches are just
> plain text, `git send-email` will do this.
>
> Thanks for the patch though! Please send any more fixes that you find :)
>
> Alistair
>
> >
> > diff --git a/target/riscv/pmp.c b/target/riscv/pmp.c
> > index d4f1007109..9308672e20 100644
> > --- a/target/riscv/pmp.c
> > +++ b/target/riscv/pmp.c
> > @@ -235,8 +235,9 @@ bool pmp_hart_has_privs(CPURISCVState *env,
> target_ulong
> > addr,
> >  /* 1.10 draft priv spec states there is an implicit order
> >   from low to high */
> >  for (i = 0; i < MAX_RISCV_PMPS; i++) {
> > +/* catch zero-size instruction checks */
> >  s = pmp_is_in_range(env, i, addr);
> > -e = pmp_is_in_range(env, i, addr + size - 1);
> > +e = pmp_is_in_range(env, i, (size == 0) ? addr : addr + size -
> 1);
> >
> >  /* partially inside */
> >  if ((s + e) == 1) {
> >
> >
>

[PATCH] target/riscv: PMP violation due to wrong size parameter

[Dayeol Lee]

riscv_cpu_tlb_fill() uses the `size` parameter to check PMP violation
using pmp_hart_has_privs().
However, if the size is unknown (=0), the ending address will be
`addr - 1` as it is `addr + size - 1` in `pmp_hart_has_privs()`.
This always causes a false PMP violation on the starting address of the
range, as `addr - 1` is not in the range.

In order to fix, we just assume that all bytes from addr to the end of
the page will be accessed if the size is unknown.

Signed-off-by: Dayeol Lee 
Reviewed-by: Richard Henderson 
---
 target/riscv/cpu_helper.c | 13 -
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/target/riscv/cpu_helper.c b/target/riscv/cpu_helper.c
index e32b6126af..7d9a22b601 100644
--- a/target/riscv/cpu_helper.c
+++ b/target/riscv/cpu_helper.c
@@ -441,6 +441,7 @@ bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address, int 
size,
 CPURISCVState *env = >env;
 hwaddr pa = 0;
 int prot;
+int pmp_size = 0;
 bool pmp_violation = false;
 int ret = TRANSLATE_FAIL;
 int mode = mmu_idx;
@@ -460,9 +461,19 @@ bool riscv_cpu_tlb_fill(CPUState *cs, vaddr address, int 
size,
   "%s address=%" VADDR_PRIx " ret %d physical " TARGET_FMT_plx
   " prot %d\n", __func__, address, ret, pa, prot);
 
+/*
+ * if size is unknown (0), assume that all bytes
+ * from addr to the end of the page will be accessed.
+ */
+if (size == 0) {
+pmp_size = -(address | TARGET_PAGE_MASK);
+} else {
+pmp_size = size;
+}
+
 if (riscv_feature(env, RISCV_FEATURE_PMP) &&
 (ret == TRANSLATE_SUCCESS) &&
-!pmp_hart_has_privs(env, pa, size, 1 << access_type, mode)) {
+!pmp_hart_has_privs(env, pa, pmp_size, 1 << access_type, mode)) {
 ret = TRANSLATE_PMP_FAIL;
 }
 if (ret == TRANSLATE_PMP_FAIL) {
-- 
2.20.1

Re: [PATCH v2 1/1] target/riscv/pmp: Fix bug preventing

[Alistair Francis]

On Sun, Oct 6, 2019 at 1:32 AM Chris Williams  wrote:
>
> Hello. I hope you don't mind me resubmitting this patch. Please let me know if

Not at all!

Thanks for the patch.

In future if people don't respond you can just keep pinging your patch.

> I've formatted it incorrectly or if it needs more explanation. My previous
> attempt probably wasn't formatted quite right. This is my first time
> contributing to Qemu, so I'm keen to get it right - thanks.

This whole paragraph should not be in the commit. It can go below the
line though.

Also please use `git format-patch` to format the patch and then `git
send-email` to send the patch. There is a whole heap of detail here:
https://wiki.qemu.org/Contribute/SubmitAPatch

>
> This fixes an issue that prevents a RISC-V CPU from executing instructions
> immediately from the base address of a PMP TOR region.
>
> When jumping to an instruction in a PMP TOR region, pmp_hart_has_privs() is
> called to validate the access. If this instruction is the very first word of a
> PMP TOR region, at address 0 relative to the start address of the region, then
> the access will fail. This is because pmp_hart_has_privs() is called with size
> 0 to perform this validation, causing this check...
>
> e = pmp_is_in_range(env, i, addr + size - 1);
>
> ... to fail, as (addr + size - 1) falls below the base address of the PMP
> region. Really, the access should succeed. For example, if I have a region
> spanning 0x80d96000 to 0x88d95fff and the CPU jumps to 0x80d96000, then:
>
> s = 0x80d96000
> e = 0x80d95fff
>
> And the validation fails. The size check proposed below catches these 
> zero-size
> instruction fetch access probes. The word alignment in pmpaddr{0-15} and
> earlier instruction alignment checks should prevent the execution of
> instructions over the upper boundary of the PMP region, though I'm happy to 
> give
> this more attention if this is a concern.

This seems like a similar issue to this patch as well:
https://lore.kernel.org/qemu-devel/20191007052813.25814-1-day...@berkeley.edu/

>From that discussion:

"In general, size 0 means "unknown size".  In this case, the one tlb lookup is
going to be used by lots of instructions -- everything that fits on the page."

Richard's last comment seems like a better fix:

"You certainly could do

if (size == 0) {
size = -(addr | TARGET_PAGE_MASK);
}

to assume that all bytes from addr to the end of the page are accessed.  That
would avoid changing too much of the rest of the logic.

That said, this code will continue to not work for mis-aligned boundaries."

So I don't think this is the correct solution. I'm not sure if Dayeol
is planning on sending a follow up version. If not feel free to send
it.

>
> Signed-off-by: Chris Williams mailto:diodes...@tuta.io>>

It looks like this is a HTML patch, also ensure all patches are just
plain text, `git send-email` will do this.

Thanks for the patch though! Please send any more fixes that you find :)

Alistair

>
> diff --git a/target/riscv/pmp.c b/target/riscv/pmp.c
> index d4f1007109..9308672e20 100644
> --- a/target/riscv/pmp.c
> +++ b/target/riscv/pmp.c
> @@ -235,8 +235,9 @@ bool pmp_hart_has_privs(CPURISCVState *env, target_ulong
> addr,
>  /* 1.10 draft priv spec states there is an implicit order
>   from low to high */
>  for (i = 0; i < MAX_RISCV_PMPS; i++) {
> +/* catch zero-size instruction checks */
>  s = pmp_is_in_range(env, i, addr);
> -e = pmp_is_in_range(env, i, addr + size - 1);
> +e = pmp_is_in_range(env, i, (size == 0) ? addr : addr + size - 1);
>
>  /* partially inside */
>  if ((s + e) == 1) {
>
>

Re: [PATCH 1/3] hw/char: Add the BCM2835 miniuart

[Alistair Francis]

On Mon, Oct 7, 2019 at 10:18 AM Philippe Mathieu-Daudé  wrote:
>
> The miniuart code is already present in the multi-device
> hw/char/bcm2835_aux.c. Simply extracting it does not generate
> patch easy to review. Instead, add it again, rename the function
> names accordingly, use the "hw/registerfields.h" API.
> This is roughtly a copy of commit 97398d900caacc.
>
> Signed-off-by: Philippe Mathieu-Daudé 

Acked-by: Alistair Francis 

Alistair

> ---
>  hw/char/Makefile.objs  |   1 +
>  hw/char/bcm2835_miniuart.c | 327 +
>  hw/char/trace-events   |   4 +
>  include/hw/char/bcm2835_miniuart.h |  37 
>  4 files changed, 369 insertions(+)
>  create mode 100644 hw/char/bcm2835_miniuart.c
>  create mode 100644 include/hw/char/bcm2835_miniuart.h
>
> diff --git a/hw/char/Makefile.objs b/hw/char/Makefile.objs
> index 02d8a66925..5bd93bde9f 100644
> --- a/hw/char/Makefile.objs
> +++ b/hw/char/Makefile.objs
> @@ -22,6 +22,7 @@ obj-$(CONFIG_DIGIC) += digic-uart.o
>  obj-$(CONFIG_STM32F2XX_USART) += stm32f2xx_usart.o
>  obj-$(CONFIG_RASPI) += bcm2835_aux.o
>
> +common-obj-$(CONFIG_RASPI) += bcm2835_miniuart.o
>  common-obj-$(CONFIG_CMSDK_APB_UART) += cmsdk-apb-uart.o
>  common-obj-$(CONFIG_ETRAXFS) += etraxfs_ser.o
>  common-obj-$(CONFIG_ISA_DEBUG) += debugcon.o
> diff --git a/hw/char/bcm2835_miniuart.c b/hw/char/bcm2835_miniuart.c
> new file mode 100644
> index 00..0e99cecce7
> --- /dev/null
> +++ b/hw/char/bcm2835_miniuart.c
> @@ -0,0 +1,327 @@
> +/*
> + * BCM2835 (Raspberry Pi) mini UART block.
> + *
> + * Copyright (c) 2015, Microsoft
> + * Written by Andrew Baumann
> + * Based on pl011.c.
> + *
> + * This code is licensed under the GPL.
> + *
> + * At present only the core UART functions (data path for tx/rx) are
> + * implemented. The following features/registers are unimplemented:
> + *  - Line/modem control
> + *  - Scratch register
> + *  - Extra control
> + *  - Baudrate
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu/log.h"
> +#include "hw/char/bcm2835_miniuart.h"
> +#include "hw/qdev-properties.h"
> +#include "hw/registerfields.h"
> +#include "migration/vmstate.h"
> +#include "trace.h"
> +
> +REG32(MU_IO,0x00)
> +REG32(MU_IER,   0x04)
> +REG32(MU_IIR,   0x08)
> +REG32(MU_LCR,   0x0c)
> +REG32(MU_MCR,   0x10)
> +REG32(MU_LSR,   0x14)
> +REG32(MU_MSR,   0x18)
> +REG32(MU_SCRATCH,   0x1c)
> +REG32(MU_CNTL,  0x20)
> +REG32(MU_STAT,  0x24)
> +REG32(MU_BAUD,  0x28)
> +
> +/* bits in IER/IIR registers */
> +#define RX_INT  0x1
> +#define TX_INT  0x2
> +
> +static void bcm2835_miniuart_update(BCM2835MiniUartState *s)
> +{
> +/*
> + * Signal an interrupt if either:
> + *
> + * 1. rx interrupt is enabled and we have a non-empty rx fifo, or
> + * 2. the tx interrupt is enabled (since we instantly drain the tx fifo)
> + */
> +s->iir = 0;
> +if ((s->ier & RX_INT) && s->read_count != 0) {
> +s->iir |= RX_INT;
> +}
> +if (s->ier & TX_INT) {
> +s->iir |= TX_INT;
> +}
> +qemu_set_irq(s->irq, s->iir != 0);
> +}
> +
> +static bool is_16650(hwaddr offset)
> +{
> +return offset < A_MU_CNTL;
> +}
> +
> +static uint64_t bcm2835_miniuart_read(void *opaque, hwaddr offset,
> +  unsigned size)
> +{
> +BCM2835MiniUartState *s = opaque;
> +uint32_t c, res = 0;
> +
> +switch (offset) {
> +case A_MU_IO:
> +/* "DLAB bit set means access baudrate register" is NYI */
> +c = s->read_fifo[s->read_pos];
> +if (s->read_count > 0) {
> +s->read_count--;
> +if (++s->read_pos == BCM2835_MINIUART_RX_FIFO_LEN) {
> +s->read_pos = 0;
> +}
> +}
> +qemu_chr_fe_accept_input(>chr);
> +bcm2835_miniuart_update(s);
> +res = c;
> +break;
> +
> +case A_MU_IER:
> +/* "DLAB bit set means access baudrate register" is NYI */
> +res = 0xc0 | s->ier; /* FIFO enables always read 1 */
> +break;
> +
> +case A_MU_IIR:
> +res = 0xc0; /* FIFO enables */
> +/*
> + * The spec is unclear on what happens when both tx and rx
> + * interrupts are active, besides that this cannot occur. At
> + * present, we choose to prioritise the rx interrupt, since
> + * the tx fifo is always empty.
> + */
> +if (s->read_count != 0) {
> +res |= 0x4;
> +} else {
> +res |= 0x2;
> +}
> +if (s->iir == 0) {
> +res |= 0x1;
> +}
> +break;
> +
> +case A_MU_LCR:
> +qemu_log_mask(LOG_UNIMP, "%s: A_MU_LCR_REG unsupported\n", __func__);
> +break;
> +
> +case A_MU_MCR:
> +qemu_log_mask(LOG_UNIMP, "%s: A_MU_MCR_REG unsupported\n", __func__);
> +break;
> +
> +case A_MU_LSR:
> +res = 0x60; /* tx idle, empty */
> +if 

Re: [PATCH 05/19] hw/arm/bcm2835: Add various unimplemented peripherals

[Alistair Francis]

On Tue, Oct 8, 2019 at 2:43 AM Philippe Mathieu-Daudé  wrote:
>
> Hi Alistair,
>
> On 9/27/19 11:42 PM, Alistair Francis wrote:
> >   On Thu, Sep 26, 2019 at 10:44 AM Philippe Mathieu-Daudé
> >  wrote:
> >>
> >> Base addresses and sizes taken from the "BCM2835 ARM Peripherals"
> >> datasheet from February 06 2012:
> >> https://www.raspberrypi.org/app/uploads/2012/02/BCM2835-ARM-Peripherals.pdf
> >>
> >> Reviewed-by: Peter Maydell 
> >> Signed-off-by: Philippe Mathieu-Daudé 
> >> ---
> >>   hw/arm/bcm2835_peripherals.c | 31 
> >>   include/hw/arm/bcm2835_peripherals.h | 15 ++
> >>   include/hw/arm/raspi_platform.h  |  8 +++
> >>   3 files changed, 54 insertions(+)
> >>
> >> diff --git a/hw/arm/bcm2835_peripherals.c b/hw/arm/bcm2835_peripherals.c
> >> index 1bd2ff41d5..fdcf616c56 100644
> >> --- a/hw/arm/bcm2835_peripherals.c
> >> +++ b/hw/arm/bcm2835_peripherals.c
> >> @@ -22,6 +22,20 @@
> >>   /* Capabilities for SD controller: no DMA, high-speed, default clocks 
> >> etc. */
> >>   #define BCM2835_SDHC_CAPAREG 0x52134b4
> >>
> >> +static void create_unimp(BCM2835PeripheralState *ps,
> >> + UnimplementedDeviceState *uds,
> >> + const char *name, hwaddr ofs, hwaddr size)
> >> +{
> >> +sysbus_init_child_obj(OBJECT(ps), name, uds,
> >> +  sizeof(UnimplementedDeviceState),
> >> +  TYPE_UNIMPLEMENTED_DEVICE);
> >> +qdev_prop_set_string(DEVICE(uds), "name", name);
> >> +qdev_prop_set_uint64(DEVICE(uds), "size", size);
> >> +object_property_set_bool(OBJECT(uds), true, "realized", _fatal);
> >> +memory_region_add_subregion_overlap(>peri_mr, ofs,
> >> +sysbus_mmio_get_region(SYS_BUS_DEVICE(uds), 0), 
> >> -1000);
> >
> > Why not just use create_unimplemented_device() and not bother saving
> > the UnimplementedDeviceState members in the struct?
>
> create_unimplemented_device() calls
>   -> sysbus_mmio_map_overlap()
>  -> sysbus_mmio_map_common()
>-> memory_region_del_subregion(get_system_memory())
>
> So it maps the device at *absolute* offset in the system memory.
>
> create_unimp() maps the device at offset *relative* to peri_mr.
>
> Patch 8 of this series maps the PERI (container) block at peri_base
> (fixed at BCM2836_PERI_BASE=0x3F00 for the 2836/2837), then patch 12
> adds the 2838 which has PERI mapped at 0xfe00. So we have the same
> "container" block mapped at different addresses.
> Not the PERI block itself doesn't know its base address, all offsets are
> relative.
>
> So using create_unimp() allow to use the same block in two different SoCs.

Ah, makes sense.

Maybe that is worth adding in the commit message, unless it's just
obvious and I am missing something.

Either way:

Reviewed-by: Alistair Francis 

Alistair

>
> 8:  https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg00678.html
> 12: https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg00684.html
>
> >> +}
> >> +
> >>   static void bcm2835_peripherals_init(Object *obj)
> >>   {
> >>   BCM2835PeripheralState *s = BCM2835_PERIPHERALS(obj);
> >> @@ -323,6 +337,23 @@ static void bcm2835_peripherals_realize(DeviceState 
> >> *dev, Error **errp)
> >>   error_propagate(errp, err);
> >>   return;
> >>   }
> >> +
> >> +create_unimp(s, >armtmr, "bcm2835-sp804", ARMCTRL_TIMER0_1_OFFSET, 
> >> 0x40);
> >> +create_unimp(s, >systmr, "bcm2835-systimer", ST_OFFSET, 0x20);
> >> +create_unimp(s, >cprman, "bcm2835-cprman", CPRMAN_OFFSET, 0x1000);
> >> +create_unimp(s, >a2w, "bcm2835-a2w", A2W_OFFSET, 0x1000);
> >> +create_unimp(s, >i2s, "bcm2835-i2s", I2S_OFFSET, 0x100);
> >> +create_unimp(s, >smi, "bcm2835-smi", SMI_OFFSET, 0x100);
> >> +create_unimp(s, >spi[0], "bcm2835-spi0", SPI0_OFFSET, 0x20);
> >> +create_unimp(s, >bscsl, "bcm2835-spis", BSC_SL_OFFSET, 0x100);
> >> +create_unimp(s, >i2c[0], "bcm2835-i2c0", BSC0_OFFSET, 0x20);
> >> +create_unimp(s, >i2c[1], "bcm2835-i2c1", BSC1_OFFSET, 0x20);
> >> +create_unimp(s, >i2c[2], "bcm2835-i2c2", BSC2_OFFSET, 0x20);
> >> +create_unimp(s, >otp, "bcm2835-otp", OTP_OFFSET, 0x80);
> >> +create_unimp(s, >dbus, "bcm2835-dbus", DBUS_OFFSET, 0x8000);
> >> +create_unimp(s, >ave0, "bcm2835-ave0", AVE0_OFFSET, 0x8000);
> >> +create_unimp(s, >dwc2, "dwc-usb2", USB_OTG_OFFSET, 0x1000);
> >> +create_unimp(s, >sdramc, "bcm2835-sdramc", SDRAMC_OFFSET, 0x100);
> >>   }
> >>
> >>   static void bcm2835_peripherals_class_init(ObjectClass *oc, void *data)
> >> diff --git a/include/hw/arm/bcm2835_peripherals.h 
> >> b/include/hw/arm/bcm2835_peripherals.h
> >> index 6b17f6a382..62a4c7b559 100644
> >> --- a/include/hw/arm/bcm2835_peripherals.h
> >> +++ b/include/hw/arm/bcm2835_peripherals.h
> >> @@ -23,6 +23,7 @@
> >>   #include "hw/sd/sdhci.h"
> >>   #include "hw/sd/bcm2835_sdhost.h"
> >>   #include "hw/gpio/bcm2835_gpio.h"
> >> 

Re: [PATCH 2/4] qemu-io: Support help options for --object

[Eric Blake]

On 10/11/19 3:55 PM, Kevin Wolf wrote:

Instead of parsing help options as normal object properties and
returning an error, provide the same help functionality as the system
emulator in qemu-io, too.

Signed-off-by: Kevin Wolf 
---
  qemu-io.c | 9 -
  1 file changed, 8 insertions(+), 1 deletion(-)



Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

ideas towards requiring VPATH build

[Eric Blake]

I know we've talked about enforcing a VPATH build, but haven't yet 
flipped the switch.  This week, I've played with using a VPATH build (cd 
qemu; mkdir -p build; cd ./build; ../configure ...; make ...), but find 
my old habits of expecting an in-tree build to just work (cd qemu; make 
...) hard to overcome.  So this is what I've come up with: if you place 
the following file in-tree, then any 'make ...' command you type in-tree 
without using -C will have the same effect as if you had typed the same 
command in the build directory, but without having to manually remember 
to switch to the build directory.


Perhaps this can be a starting point for a patch to actually include 
this file in qemu.git as part of the larger effort to force VPATH 
builds, while still having the convenience of in-tree make working for 
those who were used to it.  (I places an echo and sleep in my file to 
remind myself when I forgot to use the build directory, but that is not 
mandatory if we want GNUmakefile stored in qemu.git).


Presumably, any full switch to force a VPATH build would also include 
creating the build directory as needed (my hack assumes that it already 
exists).


$ cat GNUmakefile
# Hack for redirecting while reminding myself to use distinct builddir
%: force
@echo 'changing directory to build...'
@sleep 2
@$(MAKE) -C build -f Makefile $(MAKECMDGOALS)
force: ;
GNUmakefile: ;

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PULL 01/19] util/hbitmap: strict hbitmap_reset

[Eric Blake]

On 10/11/19 4:25 PM, John Snow wrote:

From: Vladimir Sementsov-Ogievskiy 

hbitmap_reset has an unobvious property: it rounds requested region up.
It may provoke bugs, like in recently fixed write-blocking mode of
mirror: user calls reset on unaligned region, not keeping in mind that
there are possible unrelated dirty bytes, covered by rounded-up region
and information of this unrelated "dirtiness" will be lost.

Make hbitmap_reset strict: assert that arguments are aligned, allowing
only one exception when @start + @count == hb->orig_size. It's needed
to comfort users of hbitmap_next_dirty_area, which cares about
hb->orig_size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20190806152611.280389-1-vsement...@virtuozzo.com>
[Maintainer edit: Max's suggestions from on-list. --js]
Signed-off-by: John Snow 
---
  include/qemu/hbitmap.h | 5 +
  tests/test-hbitmap.c   | 2 +-
  util/hbitmap.c | 4 
  3 files changed, 10 insertions(+), 1 deletion(-)




+++ b/util/hbitmap.c
@@ -476,6 +476,10 @@ void hbitmap_reset(HBitmap *hb, uint64_t start, uint64_t 
count)
  /* Compute range in the last layer.  */
  uint64_t first;
  uint64_t last = start + count - 1;
+uint64_t gran = 1ULL << hb->granularity;
+
+assert(!(start & (gran - 1)));
+assert(!(count & (gran - 1)) || (start + count == hb->orig_size));


I know I'm replying a bit late (since this is now a pull request), but 
would it be worth using the dedicated macro:


assert(QEMU_IS_ALIGNED(start, gran));
assert(QEMU_IS_ALIGNED(count, gran) || start + count == hb->orig_size);

instead of open-coding it?  (I would also drop the extra () around the 
right half of ||). If we want it, that would now be a followup patch.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

[PULL 16/19] block/qcow2-bitmap: fix and improve qcow2_reopen_bitmaps_rw

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

- Correct check for write access to file child, and in correct place
  (only if we want to write).
- Support reopen rw -> rw (which will be used in following commit),
  for example, !bdrv_dirty_bitmap_readonly() is not a corruption if
  bitmap is marked IN_USE in the image.
- Consider unexpected bitmap as a corruption and check other
  combinations of in-image and in-RAM bitmaps.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Message-id: 20190927122355.7344-9-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 block/qcow2-bitmap.c | 77 +---
 1 file changed, 58 insertions(+), 19 deletions(-)

diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index f7dfb40256e..98294a76965 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1108,18 +1108,14 @@ int qcow2_reopen_bitmaps_rw(BlockDriverState *bs, Error 
**errp)
 Qcow2BitmapList *bm_list;
 Qcow2Bitmap *bm;
 GSList *ro_dirty_bitmaps = NULL;
-int ret = 0;
+int ret = -EINVAL;
+bool need_header_update = false;
 
 if (s->nb_bitmaps == 0) {
 /* No bitmaps - nothing to do */
 return 0;
 }
 
-if (!can_write(bs)) {
-error_setg(errp, "Can't write to the image on reopening bitmaps rw");
-return -EINVAL;
-}
-
 bm_list = bitmap_list_load(bs, s->bitmap_directory_offset,
s->bitmap_directory_size, errp);
 if (bm_list == NULL) {
@@ -1128,32 +1124,75 @@ int qcow2_reopen_bitmaps_rw(BlockDriverState *bs, Error 
**errp)
 
 QSIMPLEQ_FOREACH(bm, bm_list, entry) {
 BdrvDirtyBitmap *bitmap = bdrv_find_dirty_bitmap(bs, bm->name);
-if (bitmap == NULL) {
-continue;
-}
 
-if (!bdrv_dirty_bitmap_readonly(bitmap)) {
-error_setg(errp, "Bitmap %s was loaded prior to rw-reopen, but was 
"
-   "not marked as readonly. This is a bug, something went "
-   "wrong. All of the bitmaps may be corrupted", bm->name);
-ret = -EINVAL;
+if (!bitmap) {
+error_setg(errp, "Unexpected bitmap '%s' in image '%s'",
+   bm->name, bs->filename);
 goto out;
 }
 
-bm->flags |= BME_FLAG_IN_USE;
-ro_dirty_bitmaps = g_slist_append(ro_dirty_bitmaps, bitmap);
+if (!(bm->flags & BME_FLAG_IN_USE)) {
+if (!bdrv_dirty_bitmap_readonly(bitmap)) {
+error_setg(errp, "Corruption: bitmap '%s' is not marked IN_USE 
"
+   "in the image '%s' and not marked readonly in RAM",
+   bm->name, bs->filename);
+goto out;
+}
+if (bdrv_dirty_bitmap_inconsistent(bitmap)) {
+error_setg(errp, "Corruption: bitmap '%s' is inconsistent but "
+   "is not marked IN_USE in the image '%s'", bm->name,
+   bs->filename);
+goto out;
+}
+
+bm->flags |= BME_FLAG_IN_USE;
+need_header_update = true;
+} else {
+/*
+ * What if flags already has BME_FLAG_IN_USE ?
+ *
+ * 1. if we are reopening RW -> RW it's OK, of course.
+ * 2. if we are reopening RO -> RW:
+ *   2.1 if @bitmap is inconsistent, it's OK. It means that it was
+ *   inconsistent (IN_USE) when we loaded it
+ *   2.2 if @bitmap is not inconsistent. This seems to be 
impossible
+ *   and implies third party interaction. Let's error-out for
+ *   safety.
+ */
+if (bdrv_dirty_bitmap_readonly(bitmap) &&
+!bdrv_dirty_bitmap_inconsistent(bitmap))
+{
+error_setg(errp, "Corruption: bitmap '%s' is marked IN_USE "
+   "in the image '%s' but it is readonly and "
+   "consistent in RAM",
+   bm->name, bs->filename);
+goto out;
+}
+}
+
+if (bdrv_dirty_bitmap_readonly(bitmap)) {
+ro_dirty_bitmaps = g_slist_append(ro_dirty_bitmaps, bitmap);
+}
 }
 
-if (ro_dirty_bitmaps != NULL) {
+if (need_header_update) {
+if (!can_write(bs->file->bs) || !(bs->file->perm & BLK_PERM_WRITE)) {
+error_setg(errp, "Failed to reopen bitmaps rw: no write access "
+   "the protocol file");
+goto out;
+}
+
 /* in_use flags must be updated */
 ret = update_ext_header_and_dir_in_place(bs, bm_list);
 if (ret < 0) {
-error_setg_errno(errp, -ret, "Can't update bitmap directory");
+error_setg_errno(errp, -ret, "Cannot update bitmap directory");
 goto out;
 }
-g_slist_foreach(ro_dirty_bitmaps, set_readonly_helper, false);
 }
 
+

Re: [PATCH 3/4] qemu-img: Support help options for --object

[Eric Blake]

On 10/11/19 3:55 PM, Kevin Wolf wrote:

Instead of parsing help options as normal object properties and
returning an error, provide the same help functionality as the system
emulator in qemu-img, too.

Signed-off-by: Kevin Wolf 
---
  qemu-img.c | 34 +-
  1 file changed, 21 insertions(+), 13 deletions(-)

Missing man page documentation of this new feature.  Perhaps iotest 
coverage is also warranted?  (qemu-io is only for testing purposes, so I 
don't care about that in patch 2 as much as I do here)


But the patch itself is a strict improvement, so

Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH 1/4] vl: Split off user_creatable_print_help()

[Eric Blake]

On 10/11/19 3:55 PM, Kevin Wolf wrote:

Printing help for --object is something that we don't only want in the


s/don't/not/


system emulator, but also in tools that support --object. Move it into a
separate function in qom/object_interfaces.c to make the code accessible
for tools.

Signed-off-by: Kevin Wolf 
---
  include/qom/object_interfaces.h | 12 +++
  qom/object_interfaces.c | 61 +
  vl.c| 52 +---
  3 files changed, 74 insertions(+), 51 deletions(-)

diff --git a/include/qom/object_interfaces.h b/include/qom/object_interfaces.h
index 682ba1d9b0..3e4e1d928b 100644
--- a/include/qom/object_interfaces.h


Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

[PULL 14/19] block/qcow2-bitmap: do not remove bitmaps on reopen-ro

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

qcow2_reopen_bitmaps_ro wants to store bitmaps and then mark them all
readonly. But the latter don't work, as
qcow2_store_persistent_dirty_bitmaps removes bitmaps after storing.
It's OK for inactivation but bad idea for reopen-ro. And this leads to
the following bug:

Assume we have persistent bitmap 'bitmap0'.
Create external snapshot
  bitmap0 is stored and therefore removed
Commit snapshot
  now we have no bitmaps
Do some writes from guest (*)
  they are not marked in bitmap
Shutdown
Start
  bitmap0 is loaded as valid, but it is actually broken! It misses
  writes (*)
Incremental backup
  it will be inconsistent

So, let's stop removing bitmaps on reopen-ro. But don't rejoice:
reopening bitmaps to rw is broken too, so the whole scenario will not
work after this patch and we can't enable corresponding test cases in
260 iotests still. Reopening bitmaps rw will be fixed in the following
patches.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Message-id: 20190927122355.7344-7-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 block/qcow2.h|  3 ++-
 block/qcow2-bitmap.c | 49 ++--
 block/qcow2.c|  2 +-
 3 files changed, 37 insertions(+), 17 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index b3398a13c20..8d293f2b64e 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -739,7 +739,8 @@ Qcow2BitmapInfoList 
*qcow2_get_bitmap_info_list(BlockDriverState *bs,
 Error **errp);
 int qcow2_reopen_bitmaps_rw(BlockDriverState *bs, Error **errp);
 int qcow2_truncate_bitmaps_check(BlockDriverState *bs, Error **errp);
-void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp);
+void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs,
+  bool release_stored, Error **errp);
 int qcow2_reopen_bitmaps_ro(BlockDriverState *bs, Error **errp);
 bool qcow2_co_can_store_new_dirty_bitmap(BlockDriverState *bs,
  const char *name,
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index ebc1afccd3d..f7dfb40256e 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1440,7 +1440,32 @@ out:
 return ret;
 }
 
-void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)
+/*
+ * qcow2_store_persistent_dirty_bitmaps
+ *
+ * Stores persistent BdrvDirtyBitmap objects.
+ *
+ * @release_stored: if true, release BdrvDirtyBitmap's after storing to the
+ * image. This is used in two cases, both via qcow2_inactivate:
+ * 1. bdrv_close: It's correct to remove bitmaps on close.
+ * 2. migration: If bitmaps are migrated through migration channel via
+ *'dirty-bitmaps' migration capability they are not handled by this code.
+ *Otherwise, it's OK to drop BdrvDirtyBitmap's and reload them on
+ *invalidation.
+ *
+ * Anyway, it's correct to remove BdrvDirtyBitmap's on inactivation, as
+ * inactivation means that we lose control on disk, and therefore on bitmaps,
+ * we should sync them and do not touch more.
+ *
+ * Contrariwise, we don't want to release any bitmaps on just reopen-to-ro,
+ * when we need to store them, as image is still under our control, and it's
+ * good to keep all the bitmaps in read-only mode. Moreover, keeping them
+ * read-only is correct because this is what would happen if we opened the node
+ * readonly to begin with, and whether we opened directly or reopened to that
+ * state shouldn't matter for the state we get afterward.
+ */
+void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs,
+  bool release_stored, Error **errp)
 {
 BdrvDirtyBitmap *bitmap;
 BDRVQcow2State *s = bs->opaque;
@@ -1551,20 +1576,14 @@ void 
qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)
 g_free(tb);
 }
 
-QSIMPLEQ_FOREACH(bm, bm_list, entry) {
-/* For safety, we remove bitmap after storing.
- * We may be here in two cases:
- * 1. bdrv_close. It's ok to drop bitmap.
- * 2. inactivation. It means migration without 'dirty-bitmaps'
- *capability, so bitmaps are not marked with
- *BdrvDirtyBitmap.migration flags. It's not bad to drop them too,
- *and reload on invalidation.
- */
-if (bm->dirty_bitmap == NULL) {
-continue;
-}
+if (release_stored) {
+QSIMPLEQ_FOREACH(bm, bm_list, entry) {
+if (bm->dirty_bitmap == NULL) {
+continue;
+}
 
-bdrv_release_dirty_bitmap(bm->dirty_bitmap);
+bdrv_release_dirty_bitmap(bm->dirty_bitmap);
+}
 }
 
 success:
@@ -1592,7 +1611,7 @@ int qcow2_reopen_bitmaps_ro(BlockDriverState *bs, Error 
**errp)
 BdrvDirtyBitmap *bitmap;
 Error *local_err = NULL;
 
-qcow2_store_persistent_dirty_bitmaps(bs, _err);
+

Re: [PATCH 4/4] qemu-nbd: Support help options for --object

[Eric Blake]

On 10/11/19 3:55 PM, Kevin Wolf wrote:

Instead of parsing help options as normal object properties and
returning an error, provide the same help functionality as the system
emulator in qemu-nbd, too.

Signed-off-by: Kevin Wolf 
---
  qemu-nbd.c | 9 -
  1 file changed, 8 insertions(+), 1 deletion(-)


Missing a change in qemu-nbd.texi for man page coverage.  But the patch 
is a strict improvement, so even if we have to add a followup patch for 
documentation, I'm okay with:


Reviewed-by: Eric Blake 

This patch touches NBD, but I'm assuming it's easier to take the series 
through your tree.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

[PULL 19/19] dirty-bitmaps: remove deprecated autoload parameter

[John Snow]

This parameter has been deprecated since 2.12.0 and is eligible for
removal. Remove this parameter as it is actually completely ignored;
let's not give false hope.

Signed-off-by: John Snow 
Reviewed-by: Eric Blake 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Message-id: 20191002232411.29968-1-js...@redhat.com
---
 qemu-deprecated.texi | 20 +++-
 qapi/block-core.json |  6 +-
 blockdev.c   |  6 --
 3 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/qemu-deprecated.texi b/qemu-deprecated.texi
index 01245e0b1c4..7239e0959da 100644
--- a/qemu-deprecated.texi
+++ b/qemu-deprecated.texi
@@ -149,11 +149,6 @@ QEMU 4.1 has three options, please migrate to one of these 
three:
 
 @section QEMU Machine Protocol (QMP) commands
 
-@subsection block-dirty-bitmap-add "autoload" parameter (since 2.12.0)
-
-"autoload" parameter is now ignored. All bitmaps are automatically loaded
-from qcow2 images.
-
 @subsection query-block result field dirty-bitmaps[i].status (since 4.0)
 
 The ``status'' field of the ``BlockDirtyInfo'' structure, returned by
@@ -356,3 +351,18 @@ existing CPU models.  Management software that needs 
runnability
 guarantees must resolve the CPU model aliases using te
 ``alias-of'' field returned by the ``query-cpu-definitions'' QMP
 command.
+
+
+@node Recently removed features
+@appendix Recently removed features
+
+What follows is a record of recently removed, formerly deprecated
+features that serves as a record for users who have encountered
+trouble after a recent upgrade.
+
+@section QEMU Machine Protocol (QMP) commands
+
+@subsection block-dirty-bitmap-add "autoload" parameter (since 4.2.0)
+
+The "autoload" parameter has been ignored since 2.12.0. All bitmaps
+are automatically loaded from qcow2 images.
diff --git a/qapi/block-core.json b/qapi/block-core.json
index e6edd641f18..e4975ece4ab 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1987,10 +1987,6 @@
 #  Qcow2 disks support persistent bitmaps. Default is false for
 #  block-dirty-bitmap-add. (Since: 2.10)
 #
-# @autoload: ignored and deprecated since 2.12.
-#Currently, all dirty tracking bitmaps are loaded from Qcow2 on
-#open.
-#
 # @disabled: the bitmap is created in the disabled state, which means that
 #it will not track drive changes. The bitmap may be enabled with
 #block-dirty-bitmap-enable. Default is false. (Since: 4.0)
@@ -1999,7 +1995,7 @@
 ##
 { 'struct': 'BlockDirtyBitmapAdd',
   'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32',
-'*persistent': 'bool', '*autoload': 'bool', '*disabled': 'bool' } }
+'*persistent': 'bool', '*disabled': 'bool' } }
 
 ##
 # @BlockDirtyBitmapMergeSource:
diff --git a/blockdev.c b/blockdev.c
index 9b6eca66430..873aba3c27f 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1966,7 +1966,6 @@ static void block_dirty_bitmap_add_prepare(BlkActionState 
*common,
 qmp_block_dirty_bitmap_add(action->node, action->name,
action->has_granularity, action->granularity,
action->has_persistent, action->persistent,
-   action->has_autoload, action->autoload,
action->has_disabled, action->disabled,
_err);
 
@@ -2858,7 +2857,6 @@ out:
 void qmp_block_dirty_bitmap_add(const char *node, const char *name,
 bool has_granularity, uint32_t granularity,
 bool has_persistent, bool persistent,
-bool has_autoload, bool autoload,
 bool has_disabled, bool disabled,
 Error **errp)
 {
@@ -2890,10 +2888,6 @@ void qmp_block_dirty_bitmap_add(const char *node, const 
char *name,
 persistent = false;
 }
 
-if (has_autoload) {
-warn_report("Autoload option is deprecated and its value is ignored");
-}
-
 if (!has_disabled) {
 disabled = false;
 }
-- 
2.21.0

[PULL 11/19] iotests: add test-case to 165 to test reopening qcow2 bitmaps to RW

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

Reopening bitmaps to RW was broken prior to previous commit. Check that
it works now.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Message-id: 20190927122355.7344-4-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 tests/qemu-iotests/165 | 57 --
 tests/qemu-iotests/165.out |  4 +--
 2 files changed, 57 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/165 b/tests/qemu-iotests/165
index 5650dc7c874..951ea011a27 100755
--- a/tests/qemu-iotests/165
+++ b/tests/qemu-iotests/165
@@ -43,10 +43,10 @@ class TestPersistentDirtyBitmap(iotests.QMPTestCase):
 os.remove(disk)
 
 def mkVm(self):
-return iotests.VM().add_drive(disk)
+return iotests.VM().add_drive(disk, opts='node-name=node0')
 
 def mkVmRo(self):
-return iotests.VM().add_drive(disk, opts='readonly=on')
+return iotests.VM().add_drive(disk, opts='readonly=on,node-name=node0')
 
 def getSha256(self):
 result = self.vm.qmp('x-debug-block-dirty-bitmap-sha256',
@@ -102,6 +102,59 @@ class TestPersistentDirtyBitmap(iotests.QMPTestCase):
 
 self.vm.shutdown()
 
+def test_reopen_rw(self):
+self.vm = self.mkVm()
+self.vm.launch()
+self.qmpAddBitmap()
+
+# Calculate hashes
+
+self.writeRegions(regions1)
+sha256_1 = self.getSha256()
+
+self.writeRegions(regions2)
+sha256_2 = self.getSha256()
+assert sha256_1 != sha256_2 # Otherwise, it's not very interesting.
+
+result = self.vm.qmp('block-dirty-bitmap-clear', node='drive0',
+ name='bitmap0')
+self.assert_qmp(result, 'return', {})
+
+# Start with regions1
+
+self.writeRegions(regions1)
+assert sha256_1 == self.getSha256()
+
+self.vm.shutdown()
+
+self.vm = self.mkVmRo()
+self.vm.launch()
+
+assert sha256_1 == self.getSha256()
+
+# Check that we are in RO mode and can't modify bitmap.
+self.writeRegions(regions2)
+assert sha256_1 == self.getSha256()
+
+# Reopen to RW
+result = self.vm.qmp('x-blockdev-reopen', **{
+'node-name': 'node0',
+'driver': iotests.imgfmt,
+'file': {
+'driver': 'file',
+'filename': disk
+},
+'read-only': False
+})
+self.assert_qmp(result, 'return', {})
+
+# Check that bitmap is reopened to RW and we can write to it.
+self.writeRegions(regions2)
+assert sha256_2 == self.getSha256()
+
+self.vm.shutdown()
+
+
 if __name__ == '__main__':
 iotests.main(supported_fmts=['qcow2'],
  supported_protocols=['file'])
diff --git a/tests/qemu-iotests/165.out b/tests/qemu-iotests/165.out
index ae1213e6f86..fbc63e62f88 100644
--- a/tests/qemu-iotests/165.out
+++ b/tests/qemu-iotests/165.out
@@ -1,5 +1,5 @@
-.
+..
 --
-Ran 1 tests
+Ran 2 tests
 
 OK
-- 
2.21.0

[PULL 18/19] MAINTAINERS: Add Vladimir as a reviewer for bitmaps

[John Snow]

I already try to make sure all bitmaps patches have been reviewed by both
Red Hat and Virtuozzo anyway, so this formalizes the arrangement.

Fam meanwhile is no longer as active, so I am removing him as a co-maintainer
simply to reflect the current practice.

Signed-off-by: John Snow 
Reviewed-by: Vladimir Sementsov-Ogievskiy 
Message-id: 20191005194448.16629-2-js...@redhat.com
---
 MAINTAINERS | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 3ca814850e0..a08c92a4162 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1816,8 +1816,8 @@ F: qapi/transaction.json
 T: git https://repo.or.cz/qemu/armbru.git block-next
 
 Dirty Bitmaps
-M: Fam Zheng 
 M: John Snow 
+R: Vladimir Sementsov-Ogievskiy 
 L: qemu-bl...@nongnu.org
 S: Supported
 F: util/hbitmap.c
@@ -1826,7 +1826,6 @@ F: include/qemu/hbitmap.h
 F: include/block/dirty-bitmap.h
 F: tests/test-hbitmap.c
 F: docs/interop/bitmaps.rst
-T: git https://github.com/famz/qemu.git bitmaps
 T: git https://github.com/jnsnow/qemu.git bitmaps
 
 Character device backends
-- 
2.21.0

[PULL 15/19] iotests: add test 260 to check bitmap life after snapshot + commit

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Message-id: 20190927122355.7344-8-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 tests/qemu-iotests/260 | 89 ++
 tests/qemu-iotests/260.out | 52 ++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 142 insertions(+)
 create mode 100755 tests/qemu-iotests/260
 create mode 100644 tests/qemu-iotests/260.out

diff --git a/tests/qemu-iotests/260 b/tests/qemu-iotests/260
new file mode 100755
index 000..4f6082c9d22
--- /dev/null
+++ b/tests/qemu-iotests/260
@@ -0,0 +1,89 @@
+#!/usr/bin/env python
+#
+# Tests for temporary external snapshot when we have bitmaps.
+#
+# Copyright (c) 2019 Virtuozzo International GmbH. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+import iotests
+from iotests import qemu_img_create, file_path, log, filter_qmp_event
+
+iotests.verify_image_format(supported_fmts=['qcow2'])
+
+base, top = file_path('base', 'top')
+size = 64 * 1024 * 3
+
+
+def print_bitmap(msg, vm):
+result = vm.qmp('query-block')['return'][0]
+if 'dirty-bitmaps' in result:
+bitmap = result['dirty-bitmaps'][0]
+log('{}: name={} dirty-clusters={}'.format(msg, bitmap['name'],
+bitmap['count'] // 64 // 1024))
+else:
+log(msg + ': not found')
+
+
+def test(persistent, restart):
+assert persistent or not restart
+log("\nTestcase {}persistent {} restart\n".format(
+'' if persistent else 'non-', 'with' if restart else 'without'))
+
+qemu_img_create('-f', iotests.imgfmt, base, str(size))
+
+vm = iotests.VM().add_drive(base)
+vm.launch()
+
+vm.qmp_log('block-dirty-bitmap-add', node='drive0', name='bitmap0',
+   persistent=persistent)
+vm.hmp_qemu_io('drive0', 'write 0 64K')
+print_bitmap('initial bitmap', vm)
+
+vm.qmp_log('blockdev-snapshot-sync', device='drive0', snapshot_file=top,
+   format=iotests.imgfmt, filters=[iotests.filter_qmp_testfiles])
+vm.hmp_qemu_io('drive0', 'write 64K 512')
+print_bitmap('check that no bitmaps are in snapshot', vm)
+
+if restart:
+log("... Restart ...")
+vm.shutdown()
+vm = iotests.VM().add_drive(top)
+vm.launch()
+
+vm.qmp_log('block-commit', device='drive0', top=top,
+   filters=[iotests.filter_qmp_testfiles])
+ev = vm.events_wait((('BLOCK_JOB_READY', None),
+ ('BLOCK_JOB_COMPLETED', None)))
+log(filter_qmp_event(ev))
+if (ev['event'] == 'BLOCK_JOB_COMPLETED'):
+vm.shutdown()
+log(vm.get_log())
+exit()
+
+vm.qmp_log('block-job-complete', device='drive0')
+ev = vm.event_wait('BLOCK_JOB_COMPLETED')
+log(filter_qmp_event(ev))
+print_bitmap('check bitmap after commit', vm)
+
+vm.hmp_qemu_io('drive0', 'write 128K 64K')
+print_bitmap('check updated bitmap', vm)
+
+vm.shutdown()
+
+
+test(persistent=False, restart=False)
+test(persistent=True, restart=False)
+test(persistent=True, restart=True)
diff --git a/tests/qemu-iotests/260.out b/tests/qemu-iotests/260.out
new file mode 100644
index 000..2f0d98d0365
--- /dev/null
+++ b/tests/qemu-iotests/260.out
@@ -0,0 +1,52 @@
+
+Testcase non-persistent without restart
+
+{"execute": "block-dirty-bitmap-add", "arguments": {"name": "bitmap0", "node": 
"drive0", "persistent": false}}
+{"return": {}}
+initial bitmap: name=bitmap0 dirty-clusters=1
+{"execute": "blockdev-snapshot-sync", "arguments": {"device": "drive0", 
"format": "qcow2", "snapshot-file": "TEST_DIR/PID-top"}}
+{"return": {}}
+check that no bitmaps are in snapshot: not found
+{"execute": "block-commit", "arguments": {"device": "drive0", "top": 
"TEST_DIR/PID-top"}}
+{"return": {}}
+{"data": {"device": "drive0", "len": 65536, "offset": 65536, "speed": 0, 
"type": "commit"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": 
"USECS", "seconds": "SECS"}}
+{"execute": "block-job-complete", "arguments": {"device": "drive0"}}
+{"return": {}}
+{"data": {"device": "drive0", "len": 65536, "offset": 65536, "speed": 0, 
"type": "commit"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": 
{"microseconds": "USECS", "seconds": "SECS"}}
+check bitmap after commit: name=bitmap0 dirty-clusters=2
+check updated bitmap: name=bitmap0 dirty-clusters=3
+
+Testcase 

[PULL 17/19] qcow2-bitmap: move bitmap reopen-rw code to qcow2_reopen_commit

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

The only reason I can imagine for this strange code at the very-end of
bdrv_reopen_commit is the fact that bs->read_only updated after
calling drv->bdrv_reopen_commit in bdrv_reopen_commit. And in the same
time, prior to previous commit, qcow2_reopen_bitmaps_rw did a wrong
check for being writable, when actually it only need writable file
child not self.

So, as it's fixed, let's move things to correct place.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Acked-by: Max Reitz 
Message-id: 20190927122355.7344-10-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 include/block/block_int.h |  6 --
 block.c   | 19 ---
 block/qcow2.c | 15 ++-
 3 files changed, 14 insertions(+), 26 deletions(-)

diff --git a/include/block/block_int.h b/include/block/block_int.h
index 1e54486ad14..9ceca23ef75 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -546,12 +546,6 @@ struct BlockDriver {
  uint64_t parent_perm, uint64_t parent_shared,
  uint64_t *nperm, uint64_t *nshared);
 
-/**
- * Bitmaps should be marked as 'IN_USE' in the image on reopening image
- * as rw. This handler should realize it. It also should unset readonly
- * field of BlockDirtyBitmap's in case of success.
- */
-int (*bdrv_reopen_bitmaps_rw)(BlockDriverState *bs, Error **errp);
 bool (*bdrv_co_can_store_new_dirty_bitmap)(BlockDriverState *bs,
const char *name,
uint32_t granularity,
diff --git a/block.c b/block.c
index c548885608d..ba09d97e0a2 100644
--- a/block.c
+++ b/block.c
@@ -3935,16 +3935,12 @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state)
 BlockDriver *drv;
 BlockDriverState *bs;
 BdrvChild *child;
-bool old_can_write, new_can_write;
 
 assert(reopen_state != NULL);
 bs = reopen_state->bs;
 drv = bs->drv;
 assert(drv != NULL);
 
-old_can_write =
-!bdrv_is_read_only(bs) && !(bdrv_get_flags(bs) & BDRV_O_INACTIVE);
-
 /* If there are any driver level actions to take */
 if (drv->bdrv_reopen_commit) {
 drv->bdrv_reopen_commit(reopen_state);
@@ -3988,21 +3984,6 @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state)
 }
 
 bdrv_refresh_limits(bs, NULL);
-
-new_can_write =
-!bdrv_is_read_only(bs) && !(bdrv_get_flags(bs) & BDRV_O_INACTIVE);
-if (!old_can_write && new_can_write && drv->bdrv_reopen_bitmaps_rw) {
-Error *local_err = NULL;
-if (drv->bdrv_reopen_bitmaps_rw(bs, _err) < 0) {
-/* This is not fatal, bitmaps just left read-only, so all following
- * writes will fail. User can remove read-only bitmaps to unblock
- * writes.
- */
-error_reportf_err(local_err,
-  "%s: Failed to make dirty bitmaps writable: ",
-  bdrv_get_node_name(bs));
-}
-}
 }
 
 /*
diff --git a/block/qcow2.c b/block/qcow2.c
index 7ab1aad9779..b652bf884e5 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1834,6 +1834,20 @@ fail:
 static void qcow2_reopen_commit(BDRVReopenState *state)
 {
 qcow2_update_options_commit(state->bs, state->opaque);
+if (state->flags & BDRV_O_RDWR) {
+Error *local_err = NULL;
+
+if (qcow2_reopen_bitmaps_rw(state->bs, _err) < 0) {
+/*
+ * This is not fatal, bitmaps just left read-only, so all following
+ * writes will fail. User can remove read-only bitmaps to unblock
+ * writes or retry reopen.
+ */
+error_reportf_err(local_err,
+  "%s: Failed to make dirty bitmaps writable: ",
+  bdrv_get_node_name(state->bs));
+}
+}
 g_free(state->opaque);
 }
 
@@ -5262,7 +5276,6 @@ BlockDriver bdrv_qcow2 = {
 .bdrv_detach_aio_context  = qcow2_detach_aio_context,
 .bdrv_attach_aio_context  = qcow2_attach_aio_context,
 
-.bdrv_reopen_bitmaps_rw = qcow2_reopen_bitmaps_rw,
 .bdrv_co_can_store_new_dirty_bitmap = qcow2_co_can_store_new_dirty_bitmap,
 .bdrv_co_remove_persistent_dirty_bitmap =
 qcow2_co_remove_persistent_dirty_bitmap,
-- 
2.21.0

[PULL 08/19] block/dirty-bitmap: refactor bdrv_dirty_bitmap_next

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

bdrv_dirty_bitmap_next is always used in same pattern. So, split it
into _next and _first, instead of combining two functions into one and
add FOR_EACH_DIRTY_BITMAP macro.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Message-id: 20190916141911.5255-5-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 include/block/dirty-bitmap.h   |  9 +++--
 block.c|  4 +---
 block/dirty-bitmap.c   | 11 +++
 block/qcow2-bitmap.c   |  8 ++--
 migration/block-dirty-bitmap.c |  4 +---
 5 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 2f9b088e11e..257f0f67046 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -96,8 +96,13 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap 
*bitmap);
 bool bdrv_dirty_bitmap_get_persistence(BdrvDirtyBitmap *bitmap);
 bool bdrv_dirty_bitmap_inconsistent(const BdrvDirtyBitmap *bitmap);
 bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs);
-BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,
-BdrvDirtyBitmap *bitmap);
+
+BdrvDirtyBitmap *bdrv_dirty_bitmap_first(BlockDriverState *bs);
+BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BdrvDirtyBitmap *bitmap);
+#define FOR_EACH_DIRTY_BITMAP(bs, bitmap) \
+for (bitmap = bdrv_dirty_bitmap_first(bs); bitmap; \
+ bitmap = bdrv_dirty_bitmap_next(bitmap))
+
 char *bdrv_dirty_bitmap_sha256(const BdrvDirtyBitmap *bitmap, Error **errp);
 int64_t bdrv_dirty_bitmap_next_zero(BdrvDirtyBitmap *bitmap, uint64_t offset,
 uint64_t bytes);
diff --git a/block.c b/block.c
index bea03cfcc92..5b5b0337acc 100644
--- a/block.c
+++ b/block.c
@@ -5363,9 +5363,7 @@ static void coroutine_fn 
bdrv_co_invalidate_cache(BlockDriverState *bs,
 }
 }
 
-for (bm = bdrv_dirty_bitmap_next(bs, NULL); bm;
- bm = bdrv_dirty_bitmap_next(bs, bm))
-{
+FOR_EACH_DIRTY_BITMAP(bs, bm) {
 bdrv_dirty_bitmap_skip_store(bm, false);
 }
 
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 4e5c87a907f..6065db80949 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -851,11 +851,14 @@ bool bdrv_has_changed_persistent_bitmaps(BlockDriverState 
*bs)
 return false;
 }
 
-BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,
-BdrvDirtyBitmap *bitmap)
+BdrvDirtyBitmap *bdrv_dirty_bitmap_first(BlockDriverState *bs)
 {
-return bitmap == NULL ? QLIST_FIRST(>dirty_bitmaps) :
-QLIST_NEXT(bitmap, list);
+return QLIST_FIRST(>dirty_bitmaps);
+}
+
+BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BdrvDirtyBitmap *bitmap)
+{
+return QLIST_NEXT(bitmap, list);
 }
 
 char *bdrv_dirty_bitmap_sha256(const BdrvDirtyBitmap *bitmap, Error **errp)
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 687087d2bc2..99812b418b8 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1488,9 +1488,7 @@ void 
qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)
 }
 
 /* check constraints and names */
-for (bitmap = bdrv_dirty_bitmap_next(bs, NULL); bitmap != NULL;
- bitmap = bdrv_dirty_bitmap_next(bs, bitmap))
-{
+FOR_EACH_DIRTY_BITMAP(bs, bitmap) {
 const char *name = bdrv_dirty_bitmap_name(bitmap);
 uint32_t granularity = bdrv_dirty_bitmap_granularity(bitmap);
 Qcow2Bitmap *bm;
@@ -1610,9 +1608,7 @@ int qcow2_reopen_bitmaps_ro(BlockDriverState *bs, Error 
**errp)
 return -EINVAL;
 }
 
-for (bitmap = bdrv_dirty_bitmap_next(bs, NULL); bitmap != NULL;
- bitmap = bdrv_dirty_bitmap_next(bs, bitmap))
-{
+FOR_EACH_DIRTY_BITMAP(bs, bitmap) {
 if (bdrv_dirty_bitmap_get_persistence(bitmap)) {
 bdrv_dirty_bitmap_set_readonly(bitmap, true);
 }
diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
index 793f249aa5b..7eafface614 100644
--- a/migration/block-dirty-bitmap.c
+++ b/migration/block-dirty-bitmap.c
@@ -283,9 +283,7 @@ static int init_dirty_bitmap_migration(void)
 for (bs = bdrv_next_all_states(NULL); bs; bs = bdrv_next_all_states(bs)) {
 const char *name = bdrv_get_device_or_node_name(bs);
 
-for (bitmap = bdrv_dirty_bitmap_next(bs, NULL); bitmap;
- bitmap = bdrv_dirty_bitmap_next(bs, bitmap))
-{
+FOR_EACH_DIRTY_BITMAP(bs, bitmap) {
 if (!bdrv_dirty_bitmap_name(bitmap)) {
 continue;
 }
-- 
2.21.0

[PULL 12/19] block/qcow2-bitmap: get rid of bdrv_has_changed_persistent_bitmaps

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

Firstly, no reason to optimize failure path. Then, function name is
ambiguous: it checks for readonly and similar things, but someone may
think that it will ignore normal bitmaps which was just unchanged, and
this is in bad relation with the fact that we should drop IN_USE flag
for unchanged bitmaps in the image.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Message-id: 20190927122355.7344-5-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 include/block/dirty-bitmap.h |  1 -
 block/dirty-bitmap.c | 12 
 block/qcow2-bitmap.c | 23 +--
 3 files changed, 13 insertions(+), 23 deletions(-)

diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 257f0f67046..958e7474fb5 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -95,7 +95,6 @@ bool bdrv_has_readonly_bitmaps(BlockDriverState *bs);
 bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap);
 bool bdrv_dirty_bitmap_get_persistence(BdrvDirtyBitmap *bitmap);
 bool bdrv_dirty_bitmap_inconsistent(const BdrvDirtyBitmap *bitmap);
-bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs);
 
 BdrvDirtyBitmap *bdrv_dirty_bitmap_first(BlockDriverState *bs);
 BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BdrvDirtyBitmap *bitmap);
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 6065db80949..4bbb251b2c9 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -839,18 +839,6 @@ bool bdrv_dirty_bitmap_inconsistent(const BdrvDirtyBitmap 
*bitmap)
 return bitmap->inconsistent;
 }
 
-bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs)
-{
-BdrvDirtyBitmap *bm;
-QLIST_FOREACH(bm, >dirty_bitmaps, list) {
-if (bm->persistent && !bm->readonly && !bm->skip_store) {
-return true;
-}
-}
-
-return false;
-}
-
 BdrvDirtyBitmap *bdrv_dirty_bitmap_first(BlockDriverState *bs)
 {
 return QLIST_FIRST(>dirty_bitmaps);
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 99812b418b8..6dfc0835485 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1464,16 +1464,7 @@ void 
qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)
 Qcow2Bitmap *bm;
 QSIMPLEQ_HEAD(, Qcow2BitmapTable) drop_tables;
 Qcow2BitmapTable *tb, *tb_next;
-
-if (!bdrv_has_changed_persistent_bitmaps(bs)) {
-/* nothing to do */
-return;
-}
-
-if (!can_write(bs)) {
-error_setg(errp, "No write access");
-return;
-}
+bool need_write = false;
 
 QSIMPLEQ_INIT(_tables);
 
@@ -1499,6 +1490,8 @@ void 
qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)
 continue;
 }
 
+need_write = true;
+
 if (check_constraints_on_bitmap(bs, name, granularity, errp) < 0) {
 error_prepend(errp, "Bitmap '%s' doesn't satisfy the constraints: 
",
   name);
@@ -1537,6 +1530,15 @@ void 
qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)
 bm->dirty_bitmap = bitmap;
 }
 
+if (!need_write) {
+goto success;
+}
+
+if (!can_write(bs)) {
+error_setg(errp, "No write access");
+goto fail;
+}
+
 /* allocate clusters and store bitmaps */
 QSIMPLEQ_FOREACH(bm, bm_list, entry) {
 if (bm->dirty_bitmap == NULL) {
@@ -1578,6 +1580,7 @@ void 
qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)
 bdrv_release_dirty_bitmap(bm->dirty_bitmap);
 }
 
+success:
 bitmap_list_free(bm_list);
 return;
 
-- 
2.21.0

[PULL 09/19] block: switch reopen queue from QSIMPLEQ to QTAILQ

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

We'll need reverse-foreach in the following commit, QTAILQ support it,
so move to QTAILQ.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-id: 20190927122355.7344-2-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 include/block/block.h |  2 +-
 block.c   | 24 
 2 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/include/block/block.h b/include/block/block.h
index 37c9de7446d..f5099435136 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -195,7 +195,7 @@ typedef struct HDGeometry {
 #define BDRV_BLOCK_EOF  0x20
 #define BDRV_BLOCK_RECURSE  0x40
 
-typedef QSIMPLEQ_HEAD(BlockReopenQueue, BlockReopenQueueEntry) 
BlockReopenQueue;
+typedef QTAILQ_HEAD(BlockReopenQueue, BlockReopenQueueEntry) BlockReopenQueue;
 
 typedef struct BDRVReopenState {
 BlockDriverState *bs;
diff --git a/block.c b/block.c
index 5b5b0337acc..aaf5d796284 100644
--- a/block.c
+++ b/block.c
@@ -1719,7 +1719,7 @@ typedef struct BlockReopenQueueEntry {
  bool prepared;
  bool perms_checked;
  BDRVReopenState state;
- QSIMPLEQ_ENTRY(BlockReopenQueueEntry) entry;
+ QTAILQ_ENTRY(BlockReopenQueueEntry) entry;
 } BlockReopenQueueEntry;
 
 /*
@@ -1732,7 +1732,7 @@ static int bdrv_reopen_get_flags(BlockReopenQueue *q, 
BlockDriverState *bs)
 BlockReopenQueueEntry *entry;
 
 if (q != NULL) {
-QSIMPLEQ_FOREACH(entry, q, entry) {
+QTAILQ_FOREACH(entry, q, entry) {
 if (entry->state.bs == bs) {
 return entry->state.flags;
 }
@@ -3249,7 +3249,7 @@ static bool bdrv_recurse_has_child(BlockDriverState *bs,
  * Adds a BlockDriverState to a simple queue for an atomic, transactional
  * reopen of multiple devices.
  *
- * bs_queue can either be an existing BlockReopenQueue that has had 
QSIMPLE_INIT
+ * bs_queue can either be an existing BlockReopenQueue that has had QTAILQ_INIT
  * already performed, or alternatively may be NULL a new BlockReopenQueue will
  * be created and initialized. This newly created BlockReopenQueue should be
  * passed back in for subsequent calls that are intended to be of the same
@@ -3290,7 +3290,7 @@ static BlockReopenQueue 
*bdrv_reopen_queue_child(BlockReopenQueue *bs_queue,
 
 if (bs_queue == NULL) {
 bs_queue = g_new0(BlockReopenQueue, 1);
-QSIMPLEQ_INIT(bs_queue);
+QTAILQ_INIT(bs_queue);
 }
 
 if (!options) {
@@ -3298,7 +3298,7 @@ static BlockReopenQueue 
*bdrv_reopen_queue_child(BlockReopenQueue *bs_queue,
 }
 
 /* Check if this BlockDriverState is already in the queue */
-QSIMPLEQ_FOREACH(bs_entry, bs_queue, entry) {
+QTAILQ_FOREACH(bs_entry, bs_queue, entry) {
 if (bs == bs_entry->state.bs) {
 break;
 }
@@ -3354,7 +3354,7 @@ static BlockReopenQueue 
*bdrv_reopen_queue_child(BlockReopenQueue *bs_queue,
 
 if (!bs_entry) {
 bs_entry = g_new0(BlockReopenQueueEntry, 1);
-QSIMPLEQ_INSERT_TAIL(bs_queue, bs_entry, entry);
+QTAILQ_INSERT_TAIL(bs_queue, bs_entry, entry);
 } else {
 qobject_unref(bs_entry->state.options);
 qobject_unref(bs_entry->state.explicit_options);
@@ -3455,7 +3455,7 @@ int bdrv_reopen_multiple(BlockReopenQueue *bs_queue, 
Error **errp)
 
 assert(bs_queue != NULL);
 
-QSIMPLEQ_FOREACH(bs_entry, bs_queue, entry) {
+QTAILQ_FOREACH(bs_entry, bs_queue, entry) {
 assert(bs_entry->state.bs->quiesce_counter > 0);
 if (bdrv_reopen_prepare(_entry->state, bs_queue, errp)) {
 goto cleanup;
@@ -3463,7 +3463,7 @@ int bdrv_reopen_multiple(BlockReopenQueue *bs_queue, 
Error **errp)
 bs_entry->prepared = true;
 }
 
-QSIMPLEQ_FOREACH(bs_entry, bs_queue, entry) {
+QTAILQ_FOREACH(bs_entry, bs_queue, entry) {
 BDRVReopenState *state = _entry->state;
 ret = bdrv_check_perm(state->bs, bs_queue, state->perm,
   state->shared_perm, NULL, NULL, errp);
@@ -3489,13 +3489,13 @@ int bdrv_reopen_multiple(BlockReopenQueue *bs_queue, 
Error **errp)
 /* If we reach this point, we have success and just need to apply the
  * changes
  */
-QSIMPLEQ_FOREACH(bs_entry, bs_queue, entry) {
+QTAILQ_FOREACH(bs_entry, bs_queue, entry) {
 bdrv_reopen_commit(_entry->state);
 }
 
 ret = 0;
 cleanup_perm:
-QSIMPLEQ_FOREACH_SAFE(bs_entry, bs_queue, entry, next) {
+QTAILQ_FOREACH_SAFE(bs_entry, bs_queue, entry, next) {
 BDRVReopenState *state = _entry->state;
 
 if (!bs_entry->perms_checked) {
@@ -3512,7 +3512,7 @@ cleanup_perm:
 }
 }
 cleanup:
-QSIMPLEQ_FOREACH_SAFE(bs_entry, bs_queue, entry, next) {
+QTAILQ_FOREACH_SAFE(bs_entry, bs_queue, entry, next) {
 if (ret) {
 if (bs_entry->prepared) {
 bdrv_reopen_abort(_entry->state);
@@ -3552,7 +3552,7 @@ static BlockReopenQueueEntry 

[PULL 13/19] block/qcow2-bitmap: drop qcow2_reopen_bitmaps_rw_hint()

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

The function is unused, drop it.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Message-id: 20190927122355.7344-6-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 block/qcow2.h|  2 --
 block/qcow2-bitmap.c | 15 +--
 2 files changed, 1 insertion(+), 16 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index 113d99bf520..b3398a13c20 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -737,8 +737,6 @@ int qcow2_check_bitmaps_refcounts(BlockDriverState *bs, 
BdrvCheckResult *res,
 bool qcow2_load_dirty_bitmaps(BlockDriverState *bs, Error **errp);
 Qcow2BitmapInfoList *qcow2_get_bitmap_info_list(BlockDriverState *bs,
 Error **errp);
-int qcow2_reopen_bitmaps_rw_hint(BlockDriverState *bs, bool *header_updated,
- Error **errp);
 int qcow2_reopen_bitmaps_rw(BlockDriverState *bs, Error **errp);
 int qcow2_truncate_bitmaps_check(BlockDriverState *bs, Error **errp);
 void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp);
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 6dfc0835485..ebc1afccd3d 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1102,8 +1102,7 @@ Qcow2BitmapInfoList 
*qcow2_get_bitmap_info_list(BlockDriverState *bs,
 return list;
 }
 
-int qcow2_reopen_bitmaps_rw_hint(BlockDriverState *bs, bool *header_updated,
- Error **errp)
+int qcow2_reopen_bitmaps_rw(BlockDriverState *bs, Error **errp)
 {
 BDRVQcow2State *s = bs->opaque;
 Qcow2BitmapList *bm_list;
@@ -,10 +1110,6 @@ int qcow2_reopen_bitmaps_rw_hint(BlockDriverState *bs, 
bool *header_updated,
 GSList *ro_dirty_bitmaps = NULL;
 int ret = 0;
 
-if (header_updated != NULL) {
-*header_updated = false;
-}
-
 if (s->nb_bitmaps == 0) {
 /* No bitmaps - nothing to do */
 return 0;
@@ -1156,9 +1151,6 @@ int qcow2_reopen_bitmaps_rw_hint(BlockDriverState *bs, 
bool *header_updated,
 error_setg_errno(errp, -ret, "Can't update bitmap directory");
 goto out;
 }
-if (header_updated != NULL) {
-*header_updated = true;
-}
 g_slist_foreach(ro_dirty_bitmaps, set_readonly_helper, false);
 }
 
@@ -1169,11 +1161,6 @@ out:
 return ret;
 }
 
-int qcow2_reopen_bitmaps_rw(BlockDriverState *bs, Error **errp)
-{
-return qcow2_reopen_bitmaps_rw_hint(bs, NULL, errp);
-}
-
 /* Checks to see if it's safe to resize bitmaps */
 int qcow2_truncate_bitmaps_check(BlockDriverState *bs, Error **errp)
 {
-- 
2.21.0

[PULL 07/19] block/dirty-bitmap: drop BdrvDirtyBitmap.mutex

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

mutex field is just a pointer to bs->dirty_bitmap_mutex, so no needs
to store it in BdrvDirtyBitmap when we have bs pointer in it (since
previous patch).

Drop mutex field. Constantly use bdrv_dirty_bitmaps_lock/unlock in
block/dirty-bitmap.c to make it more obvious that it's not per-bitmap
lock. Still, for simplicity, leave bdrv_dirty_bitmap_lock/unlock
functions as an external API.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Message-id: 20190916141911.5255-4-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 block/dirty-bitmap.c | 84 +---
 1 file changed, 41 insertions(+), 43 deletions(-)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 44453ff8241..4e5c87a907f 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -29,7 +29,6 @@
 #include "qemu/main-loop.h"
 
 struct BdrvDirtyBitmap {
-QemuMutex *mutex;
 BlockDriverState *bs;
 HBitmap *bitmap;/* Dirty bitmap implementation */
 bool busy;  /* Bitmap is busy, it can't be used via QMP */
@@ -72,12 +71,12 @@ static inline void 
bdrv_dirty_bitmaps_unlock(BlockDriverState *bs)
 
 void bdrv_dirty_bitmap_lock(BdrvDirtyBitmap *bitmap)
 {
-qemu_mutex_lock(bitmap->mutex);
+bdrv_dirty_bitmaps_lock(bitmap->bs);
 }
 
 void bdrv_dirty_bitmap_unlock(BdrvDirtyBitmap *bitmap)
 {
-qemu_mutex_unlock(bitmap->mutex);
+bdrv_dirty_bitmaps_unlock(bitmap->bs);
 }
 
 /* Called with BQL or dirty_bitmap lock taken.  */
@@ -117,7 +116,6 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState 
*bs,
 }
 bitmap = g_new0(BdrvDirtyBitmap, 1);
 bitmap->bs = bs;
-bitmap->mutex = >dirty_bitmap_mutex;
 bitmap->bitmap = hbitmap_alloc(bitmap_size, ctz32(granularity));
 bitmap->size = bitmap_size;
 bitmap->name = g_strdup(name);
@@ -151,9 +149,9 @@ static bool bdrv_dirty_bitmap_busy(const BdrvDirtyBitmap 
*bitmap)
 
 void bdrv_dirty_bitmap_set_busy(BdrvDirtyBitmap *bitmap, bool busy)
 {
-qemu_mutex_lock(bitmap->mutex);
+bdrv_dirty_bitmaps_lock(bitmap->bs);
 bitmap->busy = busy;
-qemu_mutex_unlock(bitmap->mutex);
+bdrv_dirty_bitmaps_unlock(bitmap->bs);
 }
 
 /* Called with BQL taken.  */
@@ -278,10 +276,10 @@ void bdrv_enable_dirty_bitmap_locked(BdrvDirtyBitmap 
*bitmap)
 /* Called with BQL taken. */
 void bdrv_dirty_bitmap_enable_successor(BdrvDirtyBitmap *bitmap)
 {
-assert(bitmap->mutex == bitmap->successor->mutex);
-qemu_mutex_lock(bitmap->mutex);
+assert(bitmap->bs == bitmap->successor->bs);
+bdrv_dirty_bitmaps_lock(bitmap->bs);
 bdrv_enable_dirty_bitmap_locked(bitmap->successor);
-qemu_mutex_unlock(bitmap->mutex);
+bdrv_dirty_bitmaps_unlock(bitmap->bs);
 }
 
 /* Called within bdrv_dirty_bitmap_lock..unlock and with BQL taken.  */
@@ -361,9 +359,9 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BdrvDirtyBitmap 
*parent,
 {
 BdrvDirtyBitmap *ret;
 
-qemu_mutex_lock(parent->mutex);
+bdrv_dirty_bitmaps_lock(parent->bs);
 ret = bdrv_reclaim_dirty_bitmap_locked(parent, errp);
-qemu_mutex_unlock(parent->mutex);
+bdrv_dirty_bitmaps_unlock(parent->bs);
 
 return ret;
 }
@@ -543,16 +541,16 @@ bool bdrv_can_store_new_dirty_bitmap(BlockDriverState 
*bs, const char *name,
 
 void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap)
 {
-bdrv_dirty_bitmap_lock(bitmap);
+bdrv_dirty_bitmaps_lock(bitmap->bs);
 bitmap->disabled = true;
-bdrv_dirty_bitmap_unlock(bitmap);
+bdrv_dirty_bitmaps_unlock(bitmap->bs);
 }
 
 void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap)
 {
-bdrv_dirty_bitmap_lock(bitmap);
+bdrv_dirty_bitmaps_lock(bitmap->bs);
 bdrv_enable_dirty_bitmap_locked(bitmap);
-bdrv_dirty_bitmap_unlock(bitmap);
+bdrv_dirty_bitmaps_unlock(bitmap->bs);
 }
 
 BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
@@ -593,9 +591,9 @@ bool bdrv_dirty_bitmap_get_locked(BdrvDirtyBitmap *bitmap, 
int64_t offset)
 bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset)
 {
 bool ret;
-bdrv_dirty_bitmap_lock(bitmap);
+bdrv_dirty_bitmaps_lock(bitmap->bs);
 ret = bdrv_dirty_bitmap_get_locked(bitmap, offset);
-bdrv_dirty_bitmap_unlock(bitmap);
+bdrv_dirty_bitmaps_unlock(bitmap->bs);
 
 return ret;
 }
@@ -660,9 +658,9 @@ void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
 void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int64_t offset, int64_t bytes)
 {
-bdrv_dirty_bitmap_lock(bitmap);
+bdrv_dirty_bitmaps_lock(bitmap->bs);
 bdrv_set_dirty_bitmap_locked(bitmap, offset, bytes);
-bdrv_dirty_bitmap_unlock(bitmap);
+bdrv_dirty_bitmaps_unlock(bitmap->bs);
 }
 
 /* Called within bdrv_dirty_bitmap_lock..unlock */
@@ -676,15 +674,15 @@ void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap 
*bitmap,
 void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
  

[PULL 10/19] block: reverse order for reopen commits

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

It's needed to fix reopening qcow2 with bitmaps to RW. Currently it
can't work, as qcow2 needs write access to file child, to mark bitmaps
in-image with IN_USE flag. But usually children goes after parents in
reopen queue and file child is still RO on qcow2 reopen commit. Reverse
reopen order to fix it.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Acked-by: Max Reitz 
Acked-by: John Snow 
Message-id: 20190927122355.7344-3-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 block.c | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/block.c b/block.c
index aaf5d796284..c548885608d 100644
--- a/block.c
+++ b/block.c
@@ -3486,10 +3486,16 @@ int bdrv_reopen_multiple(BlockReopenQueue *bs_queue, 
Error **errp)
 bs_entry->perms_checked = true;
 }
 
-/* If we reach this point, we have success and just need to apply the
- * changes
+/*
+ * If we reach this point, we have success and just need to apply the
+ * changes.
+ *
+ * Reverse order is used to comfort qcow2 driver: on commit it need to 
write
+ * IN_USE flag to the image, to mark bitmaps in the image as invalid. But
+ * children are usually goes after parents in reopen-queue, so go from last
+ * to first element.
  */
-QTAILQ_FOREACH(bs_entry, bs_queue, entry) {
+QTAILQ_FOREACH_REVERSE(bs_entry, bs_queue, entry) {
 bdrv_reopen_commit(_entry->state);
 }
 
-- 
2.21.0

[PULL 05/19] block/dirty-bitmap: drop meta

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

Drop meta bitmaps, as they are unused.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Message-id: 20190916141911.5255-2-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 include/block/dirty-bitmap.h |  5 
 block/dirty-bitmap.c | 46 
 2 files changed, 51 deletions(-)

diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 07503b03b53..973056778aa 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -18,9 +18,6 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState 
*bs,
   uint32_t granularity,
   const char *name,
   Error **errp);
-void bdrv_create_meta_dirty_bitmap(BdrvDirtyBitmap *bitmap,
-   int chunk_size);
-void bdrv_release_meta_dirty_bitmap(BdrvDirtyBitmap *bitmap);
 int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap,
Error **errp);
@@ -54,7 +51,6 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int64_t offset, int64_t bytes);
 void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
  int64_t offset, int64_t bytes);
-BdrvDirtyBitmapIter *bdrv_dirty_meta_iter_new(BdrvDirtyBitmap *bitmap);
 BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap);
 void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter);
 
@@ -96,7 +92,6 @@ void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
 int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter);
 void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *hbi, int64_t offset);
 int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap);
-int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap);
 void bdrv_dirty_bitmap_truncate(BlockDriverState *bs, int64_t bytes);
 bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap);
 bool bdrv_has_readonly_bitmaps(BlockDriverState *bs);
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 03e0872b972..4ecf18d5df7 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -31,7 +31,6 @@
 struct BdrvDirtyBitmap {
 QemuMutex *mutex;
 HBitmap *bitmap;/* Dirty bitmap implementation */
-HBitmap *meta;  /* Meta dirty bitmap */
 bool busy;  /* Bitmap is busy, it can't be used via QMP */
 BdrvDirtyBitmap *successor; /* Anonymous child, if any. */
 char *name; /* Optional non-empty unique ID */
@@ -127,36 +126,6 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState 
*bs,
 return bitmap;
 }
 
-/* bdrv_create_meta_dirty_bitmap
- *
- * Create a meta dirty bitmap that tracks the changes of bits in @bitmap. I.e.
- * when a dirty status bit in @bitmap is changed (either from reset to set or
- * the other way around), its respective meta dirty bitmap bit will be marked
- * dirty as well.
- *
- * @bitmap: the block dirty bitmap for which to create a meta dirty bitmap.
- * @chunk_size: how many bytes of bitmap data does each bit in the meta bitmap
- * track.
- */
-void bdrv_create_meta_dirty_bitmap(BdrvDirtyBitmap *bitmap,
-   int chunk_size)
-{
-assert(!bitmap->meta);
-qemu_mutex_lock(bitmap->mutex);
-bitmap->meta = hbitmap_create_meta(bitmap->bitmap,
-   chunk_size * BITS_PER_BYTE);
-qemu_mutex_unlock(bitmap->mutex);
-}
-
-void bdrv_release_meta_dirty_bitmap(BdrvDirtyBitmap *bitmap)
-{
-assert(bitmap->meta);
-qemu_mutex_lock(bitmap->mutex);
-hbitmap_free_meta(bitmap->bitmap);
-bitmap->meta = NULL;
-qemu_mutex_unlock(bitmap->mutex);
-}
-
 int64_t bdrv_dirty_bitmap_size(const BdrvDirtyBitmap *bitmap)
 {
 return bitmap->size;
@@ -320,7 +289,6 @@ static void 
bdrv_release_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap)
 assert(!bitmap->active_iterators);
 assert(!bdrv_dirty_bitmap_busy(bitmap));
 assert(!bdrv_dirty_bitmap_has_successor(bitmap));
-assert(!bitmap->meta);
 QLIST_REMOVE(bitmap, list);
 hbitmap_free(bitmap->bitmap);
 g_free(bitmap->name);
@@ -666,15 +634,6 @@ BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap 
*bitmap)
 return iter;
 }
 
-BdrvDirtyBitmapIter *bdrv_dirty_meta_iter_new(BdrvDirtyBitmap *bitmap)
-{
-BdrvDirtyBitmapIter *iter = g_new(BdrvDirtyBitmapIter, 1);
-hbitmap_iter_init(>hbi, bitmap->meta, 0);
-iter->bitmap = bitmap;
-bitmap->active_iterators++;
-return iter;
-}
-
 void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter)
 {
 if (!iter) {
@@ -821,11 +780,6 @@ int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap)
 return hbitmap_count(bitmap->bitmap);
 }
 
-int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
-{
-return 

[PULL 04/19] block/qcow2: proper locking on bitmap add/remove paths

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

qmp_block_dirty_bitmap_add and do_block_dirty_bitmap_remove do acquire
aio context since 0a6c86d024c52b. But this is not enough: we also must
lock qcow2 mutex when access in-image metadata. Especially it concerns
freeing qcow2 clusters.

To achieve this, move qcow2_can_store_new_dirty_bitmap and
qcow2_remove_persistent_dirty_bitmap to coroutine context.

Since we work in coroutines in correct aio context, we don't need
context acquiring in blockdev.c anymore, drop it.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Message-id: 20190920082543.23444-4-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 block/qcow2.h |  11 ++--
 include/block/block_int.h |  10 ++--
 block/dirty-bitmap.c  | 102 +++---
 block/qcow2-bitmap.c  |  24 ++---
 block/qcow2.c |   5 +-
 blockdev.c|  27 +++---
 6 files changed, 131 insertions(+), 48 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index 2ed54821635..113d99bf520 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -743,12 +743,13 @@ int qcow2_reopen_bitmaps_rw(BlockDriverState *bs, Error 
**errp);
 int qcow2_truncate_bitmaps_check(BlockDriverState *bs, Error **errp);
 void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp);
 int qcow2_reopen_bitmaps_ro(BlockDriverState *bs, Error **errp);
-bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs,
-  const char *name,
-  uint32_t granularity,
-  Error **errp);
-int qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char 
*name,
+bool qcow2_co_can_store_new_dirty_bitmap(BlockDriverState *bs,
+ const char *name,
+ uint32_t granularity,
  Error **errp);
+int qcow2_co_remove_persistent_dirty_bitmap(BlockDriverState *bs,
+const char *name,
+Error **errp);
 
 ssize_t coroutine_fn
 qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 503ac9e3cd2..1e54486ad14 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -552,13 +552,13 @@ struct BlockDriver {
  * field of BlockDirtyBitmap's in case of success.
  */
 int (*bdrv_reopen_bitmaps_rw)(BlockDriverState *bs, Error **errp);
-bool (*bdrv_can_store_new_dirty_bitmap)(BlockDriverState *bs,
-const char *name,
-uint32_t granularity,
-Error **errp);
-int (*bdrv_remove_persistent_dirty_bitmap)(BlockDriverState *bs,
+bool (*bdrv_co_can_store_new_dirty_bitmap)(BlockDriverState *bs,
const char *name,
+   uint32_t granularity,
Error **errp);
+int (*bdrv_co_remove_persistent_dirty_bitmap)(BlockDriverState *bs,
+  const char *name,
+  Error **errp);
 
 /**
  * Register/unregister a buffer for I/O. For example, when the driver is
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index d1ae2e19229..03e0872b972 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -26,6 +26,7 @@
 #include "trace.h"
 #include "block/block_int.h"
 #include "block/blockjob.h"
+#include "qemu/main-loop.h"
 
 struct BdrvDirtyBitmap {
 QemuMutex *mutex;
@@ -455,18 +456,59 @@ void bdrv_release_named_dirty_bitmaps(BlockDriverState 
*bs)
  * not fail.
  * This function doesn't release corresponding BdrvDirtyBitmap.
  */
+static int coroutine_fn
+bdrv_co_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char *name,
+   Error **errp)
+{
+if (bs->drv && bs->drv->bdrv_co_remove_persistent_dirty_bitmap) {
+return bs->drv->bdrv_co_remove_persistent_dirty_bitmap(bs, name, errp);
+}
+
+return 0;
+}
+
+typedef struct BdrvRemovePersistentDirtyBitmapCo {
+BlockDriverState *bs;
+const char *name;
+Error **errp;
+int ret;
+} BdrvRemovePersistentDirtyBitmapCo;
+
+static void coroutine_fn
+bdrv_co_remove_persistent_dirty_bitmap_entry(void *opaque)
+{
+BdrvRemovePersistentDirtyBitmapCo *s = opaque;
+
+s->ret = bdrv_co_remove_persistent_dirty_bitmap(s->bs, s->name, s->errp);
+aio_wait_kick();
+}
+
 int bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char *name,
 Error **errp)
 {
-if (bs->drv && bs->drv->bdrv_remove_persistent_dirty_bitmap) {
-return 

[PULL 03/19] block/dirty-bitmap: return int from bdrv_remove_persistent_dirty_bitmap

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

It's more comfortable to not deal with local_err.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Message-id: 20190920082543.23444-3-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 block/qcow2.h|  5 ++---
 include/block/block_int.h|  6 +++---
 include/block/dirty-bitmap.h |  5 ++---
 block/dirty-bitmap.c |  9 +
 block/qcow2-bitmap.c | 18 ++
 blockdev.c   |  7 +++
 6 files changed, 25 insertions(+), 25 deletions(-)

diff --git a/block/qcow2.h b/block/qcow2.h
index a488d761ff0..2ed54821635 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -747,9 +747,8 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs,
   const char *name,
   uint32_t granularity,
   Error **errp);
-void qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs,
-  const char *name,
-  Error **errp);
+int qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char 
*name,
+ Error **errp);
 
 ssize_t coroutine_fn
 qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 0422acdf1c4..503ac9e3cd2 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -556,9 +556,9 @@ struct BlockDriver {
 const char *name,
 uint32_t granularity,
 Error **errp);
-void (*bdrv_remove_persistent_dirty_bitmap)(BlockDriverState *bs,
-const char *name,
-Error **errp);
+int (*bdrv_remove_persistent_dirty_bitmap)(BlockDriverState *bs,
+   const char *name,
+   Error **errp);
 
 /**
  * Register/unregister a buffer for I/O. For example, when the driver is
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 4b4b731b469..07503b03b53 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -37,9 +37,8 @@ int bdrv_dirty_bitmap_check(const BdrvDirtyBitmap *bitmap, 
uint32_t flags,
 Error **errp);
 void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
 void bdrv_release_named_dirty_bitmaps(BlockDriverState *bs);
-void bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs,
- const char *name,
- Error **errp);
+int bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char *name,
+Error **errp);
 void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
 void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
 void bdrv_enable_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap);
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 8f42015db95..d1ae2e19229 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -455,13 +455,14 @@ void bdrv_release_named_dirty_bitmaps(BlockDriverState 
*bs)
  * not fail.
  * This function doesn't release corresponding BdrvDirtyBitmap.
  */
-void bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs,
- const char *name,
- Error **errp)
+int bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char *name,
+Error **errp)
 {
 if (bs->drv && bs->drv->bdrv_remove_persistent_dirty_bitmap) {
-bs->drv->bdrv_remove_persistent_dirty_bitmap(bs, name, errp);
+return bs->drv->bdrv_remove_persistent_dirty_bitmap(bs, name, errp);
 }
+
+return 0;
 }
 
 bool bdrv_can_store_new_dirty_bitmap(BlockDriverState *bs, const char *name,
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index b2487101ede..9821c1628f5 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1404,9 +1404,8 @@ static Qcow2Bitmap *find_bitmap_by_name(Qcow2BitmapList 
*bm_list,
 return NULL;
 }
 
-void qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs,
-  const char *name,
-  Error **errp)
+int qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char 
*name,
+ Error **errp)
 {
 int ret;
 BDRVQcow2State *s = bs->opaque;
@@ -1416,18 +1415,19 @@ void 
qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs,
 if (s->nb_bitmaps == 0) {
 /* Absence of the bitmap is not an error: see explanation above

[PULL 06/19] block/dirty-bitmap: add bs link

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

Add bs field to BdrvDirtyBitmap structure. Drop BlockDriverState
parameter from bitmap APIs where possible.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Message-id: 20190916141911.5255-3-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 include/block/dirty-bitmap.h   | 14 +-
 block/backup.c | 14 ++
 block/dirty-bitmap.c   | 24 
 block/mirror.c |  4 ++--
 block/qcow2-bitmap.c   |  6 +++---
 blockdev.c |  6 +++---
 migration/block-dirty-bitmap.c |  7 +++
 migration/block.c  |  4 ++--
 8 files changed, 36 insertions(+), 43 deletions(-)

diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 973056778aa..2f9b088e11e 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -18,21 +18,18 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState 
*bs,
   uint32_t granularity,
   const char *name,
   Error **errp);
-int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
-   BdrvDirtyBitmap *bitmap,
+int bdrv_dirty_bitmap_create_successor(BdrvDirtyBitmap *bitmap,
Error **errp);
-BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
-BdrvDirtyBitmap *bitmap,
+BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BdrvDirtyBitmap *bitmap,
 Error **errp);
-BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
-   BdrvDirtyBitmap *bitmap,
+BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BdrvDirtyBitmap *bitmap,
Error **errp);
 void bdrv_dirty_bitmap_enable_successor(BdrvDirtyBitmap *bitmap);
 BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
 const char *name);
 int bdrv_dirty_bitmap_check(const BdrvDirtyBitmap *bitmap, uint32_t flags,
 Error **errp);
-void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
+void bdrv_release_dirty_bitmap(BdrvDirtyBitmap *bitmap);
 void bdrv_release_named_dirty_bitmaps(BlockDriverState *bs);
 int bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char *name,
 Error **errp);
@@ -106,8 +103,7 @@ int64_t bdrv_dirty_bitmap_next_zero(BdrvDirtyBitmap 
*bitmap, uint64_t offset,
 uint64_t bytes);
 bool bdrv_dirty_bitmap_next_dirty_area(BdrvDirtyBitmap *bitmap,
uint64_t *offset, uint64_t *bytes);
-BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap_locked(BlockDriverState *bs,
-  BdrvDirtyBitmap *bitmap,
+BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
   Error **errp);
 
 #endif
diff --git a/block/backup.c b/block/backup.c
index 763f0d7ff6d..acb67da3a7b 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -352,7 +352,6 @@ static int coroutine_fn backup_before_write_notify(
 static void backup_cleanup_sync_bitmap(BackupBlockJob *job, int ret)
 {
 BdrvDirtyBitmap *bm;
-BlockDriverState *bs = blk_bs(job->common.blk);
 bool sync = (((ret == 0) || (job->bitmap_mode == BITMAP_SYNC_MODE_ALWAYS)) 
\
  && (job->bitmap_mode != BITMAP_SYNC_MODE_NEVER));
 
@@ -361,13 +360,13 @@ static void backup_cleanup_sync_bitmap(BackupBlockJob 
*job, int ret)
  * We succeeded, or we always intended to sync the bitmap.
  * Delete this bitmap and install the child.
  */
-bm = bdrv_dirty_bitmap_abdicate(bs, job->sync_bitmap, NULL);
+bm = bdrv_dirty_bitmap_abdicate(job->sync_bitmap, NULL);
 } else {
 /*
  * We failed, or we never intended to sync the bitmap anyway.
  * Merge the successor back into the parent, keeping all data.
  */
-bm = bdrv_reclaim_dirty_bitmap(bs, job->sync_bitmap, NULL);
+bm = bdrv_reclaim_dirty_bitmap(job->sync_bitmap, NULL);
 }
 
 assert(bm);
@@ -398,10 +397,9 @@ static void backup_abort(Job *job)
 static void backup_clean(Job *job)
 {
 BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
-BlockDriverState *bs = blk_bs(s->common.blk);
 
 if (s->copy_bitmap) {
-bdrv_release_dirty_bitmap(bs, s->copy_bitmap);
+bdrv_release_dirty_bitmap(s->copy_bitmap);
 s->copy_bitmap = NULL;
 }
 
@@ -679,7 +677,7 @@ BlockJob *backup_job_create(const char *job_id, 
BlockDriverState *bs,
 }
 
 /* Create a new bitmap, and freeze/disable this one. */
-if 

[PULL 01/19] util/hbitmap: strict hbitmap_reset

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

hbitmap_reset has an unobvious property: it rounds requested region up.
It may provoke bugs, like in recently fixed write-blocking mode of
mirror: user calls reset on unaligned region, not keeping in mind that
there are possible unrelated dirty bytes, covered by rounded-up region
and information of this unrelated "dirtiness" will be lost.

Make hbitmap_reset strict: assert that arguments are aligned, allowing
only one exception when @start + @count == hb->orig_size. It's needed
to comfort users of hbitmap_next_dirty_area, which cares about
hb->orig_size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: Max Reitz 
Message-Id: <20190806152611.280389-1-vsement...@virtuozzo.com>
[Maintainer edit: Max's suggestions from on-list. --js]
Signed-off-by: John Snow 
---
 include/qemu/hbitmap.h | 5 +
 tests/test-hbitmap.c   | 2 +-
 util/hbitmap.c | 4 
 3 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index 4afbe6292e3..1bf944ca3d1 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -132,6 +132,11 @@ void hbitmap_set(HBitmap *hb, uint64_t start, uint64_t 
count);
  * @count: Number of bits to reset.
  *
  * Reset a consecutive range of bits in an HBitmap.
+ * @start and @count must be aligned to bitmap granularity. The only exception
+ * is resetting the tail of the bitmap: @count may be equal to hb->orig_size -
+ * @start, in this case @count may be not aligned. The sum of @start + @count 
is
+ * allowed to be greater than hb->orig_size, but only if @start < hb->orig_size
+ * and @start + @count = ALIGN_UP(hb->orig_size, granularity).
  */
 void hbitmap_reset(HBitmap *hb, uint64_t start, uint64_t count);
 
diff --git a/tests/test-hbitmap.c b/tests/test-hbitmap.c
index eed5d288cbc..e1f867085f4 100644
--- a/tests/test-hbitmap.c
+++ b/tests/test-hbitmap.c
@@ -423,7 +423,7 @@ static void test_hbitmap_granularity(TestHBitmapData *data,
 hbitmap_test_check(data, 0);
 hbitmap_test_set(data, 0, 3);
 g_assert_cmpint(hbitmap_count(data->hb), ==, 4);
-hbitmap_test_reset(data, 0, 1);
+hbitmap_test_reset(data, 0, 2);
 g_assert_cmpint(hbitmap_count(data->hb), ==, 2);
 }
 
diff --git a/util/hbitmap.c b/util/hbitmap.c
index fd44c897ab0..757d39e360a 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -476,6 +476,10 @@ void hbitmap_reset(HBitmap *hb, uint64_t start, uint64_t 
count)
 /* Compute range in the last layer.  */
 uint64_t first;
 uint64_t last = start + count - 1;
+uint64_t gran = 1ULL << hb->granularity;
+
+assert(!(start & (gran - 1)));
+assert(!(count & (gran - 1)) || (start + count == hb->orig_size));
 
 trace_hbitmap_reset(hb, start, count,
 start >> hb->granularity, last >> hb->granularity);
-- 
2.21.0

[PULL 00/19] Bitmaps patches

[John Snow]

The following changes since commit 98b2e3c9ab3abfe476a2b02f8f51813edb90e72d:

  Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into 
staging (2019-10-08 16:08:35 +0100)

are available in the Git repository at:

  https://github.com/jnsnow/qemu.git tags/bitmaps-pull-request

for you to fetch changes up to b97d9a1014b61dd0980e7f4a0c9ca1e3b0aaa761:

  dirty-bitmaps: remove deprecated autoload parameter (2019-10-09 17:02:45 
-0400)


Pull request



John Snow (2):
  MAINTAINERS: Add Vladimir as a reviewer for bitmaps
  dirty-bitmaps: remove deprecated autoload parameter

Vladimir Sementsov-Ogievskiy (17):
  util/hbitmap: strict hbitmap_reset
  block: move bdrv_can_store_new_dirty_bitmap to block/dirty-bitmap.c
  block/dirty-bitmap: return int from
bdrv_remove_persistent_dirty_bitmap
  block/qcow2: proper locking on bitmap add/remove paths
  block/dirty-bitmap: drop meta
  block/dirty-bitmap: add bs link
  block/dirty-bitmap: drop BdrvDirtyBitmap.mutex
  block/dirty-bitmap: refactor bdrv_dirty_bitmap_next
  block: switch reopen queue from QSIMPLEQ to QTAILQ
  block: reverse order for reopen commits
  iotests: add test-case to 165 to test reopening qcow2 bitmaps to RW
  block/qcow2-bitmap: get rid of bdrv_has_changed_persistent_bitmaps
  block/qcow2-bitmap: drop qcow2_reopen_bitmaps_rw_hint()
  block/qcow2-bitmap: do not remove bitmaps on reopen-ro
  iotests: add test 260 to check bitmap life after snapshot + commit
  block/qcow2-bitmap: fix and improve qcow2_reopen_bitmaps_rw
  qcow2-bitmap: move bitmap reopen-rw code to qcow2_reopen_commit

 qemu-deprecated.texi   |  20 ++-
 qapi/block-core.json   |   6 +-
 block/qcow2.h  |  19 +--
 include/block/block.h  |   2 +-
 include/block/block_int.h  |  20 +--
 include/block/dirty-bitmap.h   |  34 ++--
 include/qemu/hbitmap.h |   5 +
 block.c|  79 +++--
 block/backup.c |  14 +-
 block/dirty-bitmap.c   | 290 +++--
 block/mirror.c |   4 +-
 block/qcow2-bitmap.c   | 212 +++-
 block/qcow2.c  |  22 ++-
 blockdev.c |  40 ++---
 migration/block-dirty-bitmap.c |  11 +-
 migration/block.c  |   4 +-
 tests/test-hbitmap.c   |   2 +-
 util/hbitmap.c |   4 +
 MAINTAINERS|   3 +-
 tests/qemu-iotests/165 |  57 ++-
 tests/qemu-iotests/165.out |   4 +-
 tests/qemu-iotests/260 |  89 ++
 tests/qemu-iotests/260.out |  52 ++
 tests/qemu-iotests/group   |   1 +
 24 files changed, 624 insertions(+), 370 deletions(-)
 create mode 100755 tests/qemu-iotests/260
 create mode 100644 tests/qemu-iotests/260.out

-- 
2.21.0

[PULL 02/19] block: move bdrv_can_store_new_dirty_bitmap to block/dirty-bitmap.c

[John Snow]

From: Vladimir Sementsov-Ogievskiy 

block/dirty-bitmap.c seems to be more appropriate for it and
bdrv_remove_persistent_dirty_bitmap already in it.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Reviewed-by: John Snow 
Message-id: 20190920082543.23444-2-vsement...@virtuozzo.com
Signed-off-by: John Snow 
---
 block.c  | 22 --
 block/dirty-bitmap.c | 22 ++
 2 files changed, 22 insertions(+), 22 deletions(-)

diff --git a/block.c b/block.c
index 59441248451..bea03cfcc92 100644
--- a/block.c
+++ b/block.c
@@ -6555,25 +6555,3 @@ void bdrv_del_child(BlockDriverState *parent_bs, 
BdrvChild *child, Error **errp)
 
 parent_bs->drv->bdrv_del_child(parent_bs, child, errp);
 }
-
-bool bdrv_can_store_new_dirty_bitmap(BlockDriverState *bs, const char *name,
- uint32_t granularity, Error **errp)
-{
-BlockDriver *drv = bs->drv;
-
-if (!drv) {
-error_setg_errno(errp, ENOMEDIUM,
- "Can't store persistent bitmaps to %s",
- bdrv_get_device_or_node_name(bs));
-return false;
-}
-
-if (!drv->bdrv_can_store_new_dirty_bitmap) {
-error_setg_errno(errp, ENOTSUP,
- "Can't store persistent bitmaps to %s",
- bdrv_get_device_or_node_name(bs));
-return false;
-}
-
-return drv->bdrv_can_store_new_dirty_bitmap(bs, name, granularity, errp);
-}
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 134e0c9a0c8..8f42015db95 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -464,6 +464,28 @@ void bdrv_remove_persistent_dirty_bitmap(BlockDriverState 
*bs,
 }
 }
 
+bool bdrv_can_store_new_dirty_bitmap(BlockDriverState *bs, const char *name,
+ uint32_t granularity, Error **errp)
+{
+BlockDriver *drv = bs->drv;
+
+if (!drv) {
+error_setg_errno(errp, ENOMEDIUM,
+ "Can't store persistent bitmaps to %s",
+ bdrv_get_device_or_node_name(bs));
+return false;
+}
+
+if (!drv->bdrv_can_store_new_dirty_bitmap) {
+error_setg_errno(errp, ENOTSUP,
+ "Can't store persistent bitmaps to %s",
+ bdrv_get_device_or_node_name(bs));
+return false;
+}
+
+return drv->bdrv_can_store_new_dirty_bitmap(bs, name, granularity, errp);
+}
+
 void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap)
 {
 bdrv_dirty_bitmap_lock(bitmap);
-- 
2.21.0

Re: [Qemu-devel] [PATCH v30 0/8] QEMU AVR 8 bit cores

[Aleksandar Markovic]

On Friday, October 11, 2019, Alex Bennée  wrote:

>
> Aleksandar Markovic  writes:
>
> > On Friday, October 11, 2019, Philippe Mathieu-Daudé 
> > wrote:
> >
> >> Hi Michael,
> >>
> >> On 9/2/19 4:01 PM, Michael Rolnik wrote:
> >>
> >>> This series of patches adds 8bit AVR cores to QEMU.
> >>> All instruction, except BREAK/DES/SPM/SPMX, are implemented. Not fully
> >>> tested yet.
> >>> However I was able to execute simple code with functions. e.g fibonacci
> >>> calculation.
> >>> This series of patches include a non real, sample board.
> >>> No fuses support yet. PC is set to 0 at reset.
> >>>
> >>> the patches include the following
> >>> 1. just a basic 8bit AVR CPU, without instruction decoding or
> translation
> >>> 2. CPU features which allow define the following 8bit AVR cores
> >>>   avr1
> >>>   avr2 avr25
> >>>   avr3 avr31 avr35
> >>>   avr4
> >>>   avr5 avr51
> >>>   avr6
> >>>   xmega2 xmega4 xmega5 xmega6 xmega7
> >>> 3. a definition of sample machine with SRAM, FLASH and CPU which allows
> >>> to execute simple code
> >>> 4. encoding for all AVR instructions
> >>> 5. interrupt handling
> >>> 6. helpers for IN, OUT, SLEEP, WBR & unsupported instructions
> >>> 7. a decoder which given an opcode decides what istruction it is
> >>> 8. translation of AVR instruction into TCG
> >>> 9. all features together
> >>>
> >>> [..]
> >>
> >>> Michael Rolnik (7):
> >>>target/avr: Add outward facing interfaces and core CPU logic
> >>>target/avr: Add instruction helpers
> >>>target/avr: Add instruction decoding
> >>>target/avr: Add instruction translation
> >>>target/avr: Add example board configuration
> >>>target/avr: Register AVR support with the rest of QEMU, the build
> >>>  system, and the MAINTAINERS file
> >>>target/avr: Add tests
> >>>
> >>> Sarah Harris (1):
> >>>target/avr: Add limited support for USART and 16 bit timer
> peripherals
> >>>
> >>
> >> Overall architecture patches look good, but I'd like some more time to
> >> review the hardware patches. Unfortunately I won't have time until
> November.
> >> There was a chat on IRC about your series,
> >>
> > I don't see the reason why do you initiate IRC communication on this
> topic,
> > if we have the mailing list for discussing such important issues as
> > introducing a new target (that should be definitely visible to all
> > participants).
>
> IRC is often a good way of quickly discussing something when someone is
> about (often as a tangent from another discussion). I don't think there
> is anything wrong with that as long as it's followed up on the mailing
> list.
>
>
OK, at least the series got some attention, be it on IRC or on the list. I
still find it odd that suddenly, after months and months of this series
practically sitting on the list, any suggestion couldn't first be discussed
here, so that we collectively find the best outcome. But, yes, I probably
produced much ado about nothing. I hope that this would soon result in
helping Michael complete the integration. Micheal, whatever I said
regarding patch 4, is only a suggestion - if others are fine with it, I am
fine too. Best luck to all involved! :)

Aleksandar



> >
> >> I suggested Richard we could merge patches 1-4 and 7. They are almost
> >> sufficient to run the qemu-avr-tests gdbstub tests (but not the FreeRTOS
> >> ones).
>
> Which is was ;-)
>
> --
> Alex Bennée
>
>

[PATCH 4/4] qemu-nbd: Support help options for --object

[Kevin Wolf]

Instead of parsing help options as normal object properties and
returning an error, provide the same help functionality as the system
emulator in qemu-nbd, too.

Signed-off-by: Kevin Wolf 
---
 qemu-nbd.c | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/qemu-nbd.c b/qemu-nbd.c
index 9032b6de2a..caacf0ed73 100644
--- a/qemu-nbd.c
+++ b/qemu-nbd.c
@@ -507,6 +507,13 @@ static QemuOptsList qemu_object_opts = {
 },
 };
 
+static bool qemu_nbd_object_print_help(const char *type, QemuOpts *opts)
+{
+if (user_creatable_print_help(type, opts)) {
+exit(0);
+}
+return true;
+}
 
 
 static QCryptoTLSCreds *nbd_get_tls_creds(const char *id, bool list,
@@ -902,7 +909,7 @@ int main(int argc, char **argv)
 
 qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal);
+  qemu_nbd_object_print_help, _fatal);
 
 if (!trace_init_backends()) {
 exit(1);
-- 
2.20.1

[PATCH 2/4] qemu-io: Support help options for --object

[Kevin Wolf]

Instead of parsing help options as normal object properties and
returning an error, provide the same help functionality as the system
emulator in qemu-io, too.

Signed-off-by: Kevin Wolf 
---
 qemu-io.c | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/qemu-io.c b/qemu-io.c
index f64eca6940..91e3276592 100644
--- a/qemu-io.c
+++ b/qemu-io.c
@@ -475,6 +475,13 @@ static QemuOptsList qemu_object_opts = {
 },
 };
 
+static bool qemu_io_object_print_help(const char *type, QemuOpts *opts)
+{
+if (user_creatable_print_help(type, opts)) {
+exit(0);
+}
+return true;
+}
 
 static QemuOptsList file_opts = {
 .name = "file",
@@ -622,7 +629,7 @@ int main(int argc, char **argv)
 
 qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal);
+  qemu_io_object_print_help, _fatal);
 
 if (!trace_init_backends()) {
 exit(1);
-- 
2.20.1

[PATCH 3/4] qemu-img: Support help options for --object

[Kevin Wolf]

Instead of parsing help options as normal object properties and
returning an error, provide the same help functionality as the system
emulator in qemu-img, too.

Signed-off-by: Kevin Wolf 
---
 qemu-img.c | 34 +-
 1 file changed, 21 insertions(+), 13 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index 384c6f38bc..8b03ef8171 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -214,6 +214,14 @@ static QemuOptsList qemu_object_opts = {
 },
 };
 
+static bool qemu_img_object_print_help(const char *type, QemuOpts *opts)
+{
+if (user_creatable_print_help(type, opts)) {
+exit(0);
+}
+return true;
+}
+
 static QemuOptsList qemu_source_opts = {
 .name = "source",
 .implied_opt_name = "file",
@@ -516,7 +524,7 @@ static int img_create(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 goto fail;
 }
 
@@ -766,7 +774,7 @@ static int img_check(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 return 1;
 }
 
@@ -979,7 +987,7 @@ static int img_commit(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 return 1;
 }
 
@@ -1362,7 +1370,7 @@ static int img_compare(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 ret = 2;
 goto out4;
 }
@@ -2210,7 +2218,7 @@ static int img_convert(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 goto fail_getopt;
 }
 
@@ -2776,7 +2784,7 @@ static int img_info(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 return 1;
 }
 
@@ -3002,7 +3010,7 @@ static int img_map(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 return 1;
 }
 
@@ -3154,7 +3162,7 @@ static int img_snapshot(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 return 1;
 }
 
@@ -3321,7 +3329,7 @@ static int img_rebase(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 return 1;
 }
 
@@ -3742,7 +3750,7 @@ static int img_resize(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 return 1;
 }
 
@@ -3986,7 +3994,7 @@ static int img_amend(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 ret = -1;
 goto out_no_progress;
 }
@@ -4630,7 +4638,7 @@ static int img_dd(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 ret = -1;
 goto out;
 }
@@ -4907,7 +4915,7 @@ static int img_measure(int argc, char **argv)
 
 if (qemu_opts_foreach(_object_opts,
   user_creatable_add_opts_foreach,
-  NULL, _fatal)) {
+  qemu_img_object_print_help, _fatal)) {
 goto out;
 }
 
-- 
2.20.1

[PATCH 0/4] qemu-img/io/nbd: Support help options for --object

[Kevin Wolf]

This series fixes that the tools don't consider help options for
--object and just treat them as regular object properties. For example:

$ ./qemu-io --object secret,help
qemu-io: Parameter 'id' is missing

With these patches, we get the expected behaviour like in the system
emulator.

Kevin Wolf (4):
  vl: Split off user_creatable_print_help()
  qemu-io: Support help options for --object
  qemu-img: Support help options for --object
  qemu-nbd: Support help options for --object

 include/qom/object_interfaces.h | 12 +++
 qemu-img.c  | 34 +++---
 qemu-io.c   |  9 -
 qemu-nbd.c  |  9 -
 qom/object_interfaces.c | 61 +
 vl.c| 52 +---
 6 files changed, 111 insertions(+), 66 deletions(-)

-- 
2.20.1

[PATCH 1/4] vl: Split off user_creatable_print_help()

[Kevin Wolf]

Printing help for --object is something that we don't only want in the
system emulator, but also in tools that support --object. Move it into a
separate function in qom/object_interfaces.c to make the code accessible
for tools.

Signed-off-by: Kevin Wolf 
---
 include/qom/object_interfaces.h | 12 +++
 qom/object_interfaces.c | 61 +
 vl.c| 52 +---
 3 files changed, 74 insertions(+), 51 deletions(-)

diff --git a/include/qom/object_interfaces.h b/include/qom/object_interfaces.h
index 682ba1d9b0..3e4e1d928b 100644
--- a/include/qom/object_interfaces.h
+++ b/include/qom/object_interfaces.h
@@ -132,6 +132,18 @@ typedef bool (*user_creatable_add_opts_predicate)(const 
char *type);
 int user_creatable_add_opts_foreach(void *opaque,
 QemuOpts *opts, Error **errp);
 
+/**
+ * user_creatable_print_help:
+ * @type: the QOM type to be added
+ * @opts: options to create
+ *
+ * Prints help if requested in @opts.
+ *
+ * Returns: true if @opts contained a help option and help was printed, false
+ * if no help option was found.
+ */
+bool user_creatable_print_help(const char *type, QemuOpts *opts);
+
 /**
  * user_creatable_del:
  * @id: the unique ID for the object
diff --git a/qom/object_interfaces.c b/qom/object_interfaces.c
index cb5809934a..46cd6eab5c 100644
--- a/qom/object_interfaces.c
+++ b/qom/object_interfaces.c
@@ -1,8 +1,11 @@
 #include "qemu/osdep.h"
+
+#include "qemu/cutils.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qerror.h"
 #include "qom/object_interfaces.h"
+#include "qemu/help_option.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
 #include "qapi/opts-visitor.h"
@@ -155,6 +158,64 @@ int user_creatable_add_opts_foreach(void *opaque, QemuOpts 
*opts, Error **errp)
 return 0;
 }
 
+bool user_creatable_print_help(const char *type, QemuOpts *opts)
+{
+ObjectClass *klass;
+
+if (is_help_option(type)) {
+GSList *l, *list;
+
+printf("List of user creatable objects:\n");
+list = object_class_get_list_sorted(TYPE_USER_CREATABLE, false);
+for (l = list; l != NULL; l = l->next) {
+ObjectClass *oc = OBJECT_CLASS(l->data);
+printf("  %s\n", object_class_get_name(oc));
+}
+g_slist_free(list);
+return true;
+}
+
+klass = object_class_by_name(type);
+if (klass && qemu_opt_has_help_opt(opts)) {
+ObjectPropertyIterator iter;
+ObjectProperty *prop;
+GPtrArray *array = g_ptr_array_new();
+int i;
+
+object_class_property_iter_init(, klass);
+while ((prop = object_property_iter_next())) {
+GString *str;
+
+if (!prop->set) {
+continue;
+}
+
+str = g_string_new(NULL);
+g_string_append_printf(str, "  %s=<%s>", prop->name, prop->type);
+if (prop->description) {
+if (str->len < 24) {
+g_string_append_printf(str, "%*s", 24 - (int)str->len, "");
+}
+g_string_append_printf(str, " - %s", prop->description);
+}
+g_ptr_array_add(array, g_string_free(str, false));
+}
+g_ptr_array_sort(array, (GCompareFunc)qemu_pstrcmp0);
+if (array->len > 0) {
+printf("%s options:\n", type);
+} else {
+printf("There are no options for %s.\n", type);
+}
+for (i = 0; i < array->len; i++) {
+printf("%s\n", (char *)array->pdata[i]);
+}
+g_ptr_array_set_free_func(array, g_free);
+g_ptr_array_free(array, true);
+return true;
+}
+
+return false;
+}
 
 void user_creatable_del(const char *id, Error **errp)
 {
diff --git a/vl.c b/vl.c
index 002bf4919e..b86d4e502d 100644
--- a/vl.c
+++ b/vl.c
@@ -2649,57 +2649,7 @@ static int machine_set_property(void *opaque,
  */
 static bool object_create_initial(const char *type, QemuOpts *opts)
 {
-ObjectClass *klass;
-
-if (is_help_option(type)) {
-GSList *l, *list;
-
-printf("List of user creatable objects:\n");
-list = object_class_get_list_sorted(TYPE_USER_CREATABLE, false);
-for (l = list; l != NULL; l = l->next) {
-ObjectClass *oc = OBJECT_CLASS(l->data);
-printf("  %s\n", object_class_get_name(oc));
-}
-g_slist_free(list);
-exit(0);
-}
-
-klass = object_class_by_name(type);
-if (klass && qemu_opt_has_help_opt(opts)) {
-ObjectPropertyIterator iter;
-ObjectProperty *prop;
-GPtrArray *array = g_ptr_array_new();
-int i;
-
-object_class_property_iter_init(, klass);
-while ((prop = object_property_iter_next())) {
-GString *str;
-
-if (!prop->set) {
-continue;
-}
-
-str = g_string_new(NULL);
-

Re: [PATCH] qcow2-bitmaps: fix qcow2_can_store_new_dirty_bitmap

[John Snow]

On 10/11/19 6:02 AM, Vladimir Sementsov-Ogievskiy wrote:
> 10.10.2019 21:46, John Snow wrote:
>>
>>
>> On 10/10/19 11:39 AM, Eric Blake wrote:
>>> On 6/7/19 1:53 PM, Vladimir Sementsov-Ogievskiy wrote:
 07.06.2019 21:48, Vladimir Sementsov-Ogievskiy wrote:
> qcow2_can_store_new_dirty_bitmap works wrong, as it considers only
> bitmaps already stored in the qcow2 image and ignores persistent
> BdrvDirtyBitmap objects.
>
> So, let's instead count persistent BdrvDirtyBitmaps. We load all qcow2
> bitmaps on open, so there should not be any bitmap in the image for
> which we don't have BdrvDirtyBitmaps version. If it is - it's a kind of
> corruption, and no reason to check for corruptions here (open() and
> close() are better places for it).
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
>
> Hi!
>
> Patch is better than discussing I thing, so here is my
> counter-suggestion for
> "[PATCH 0/5] block/dirty-bitmap: check number and size constraints
> against queued bitmaps"
> by John.
>
> It's of course needs some additional refactoring, as first assert
> shows bad design,
> I just wrote it in such manner to make minimum changes to fix the bug.
>
>>>
> +    assert(!bdrv_find_dirty_bitmap(bs, name));

 exactly bad, this should be checked in qmp_block_dirty_bitmap_add(),
 before checks around
 persistence. and aio_context_acquire may be dropped..

 But anyway, idea is clear I think, consider it as RFC patch
>>>
>>> Are you planning to post a v2, as this affects at least
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1712636
>>>
>>
>> I suppose we ought to do it this way for now as it's less SLOC than my
>> idea, but I have to admit I would still prefer to get rid of "can_store"
>> altogether and provide concrete bitmap_store() and bitmap_remove()
>> callbacks for purpose of symmetry and model simplicity.
>>
>> I guess I'll worry about that discussion some other time.
>>
>> --js
>>
> 
> Should I base it on master or on you bitmaps branch? Do we want it for stable?
> 

Not sure. I'm going to send the PR out today and you can decide what's
best to do. (Please do CC qemu-stable, though.)

--js

Re: [PATCH v2 1/1] IDE: deprecate ide-drive

[John Snow]

On 10/11/19 5:12 AM, Peter Krempa wrote:
> On Thu, Oct 10, 2019 at 14:08:12 -0400, John Snow wrote:
>> On 10/10/19 7:54 AM, Peter Krempa wrote:
>>> On Thu, Oct 10, 2019 at 13:42:26 +0200, Philippe Mathieu-Daudé wrote:
 On 10/10/19 1:26 PM, Peter Krempa wrote:
> 
> [...]
> 
>>> To be honest, I didn't really review the code nor the documentation.
>>> I actually reviewed only the idea itself in the context of integration
>>> with libvirt and that's why I didn't go for 'Reviewed-by:'.
>>>
>>> The gist of the citation above is that we should stick to well known
>>> tags with their well known meanings and I think that considering this a
>>> 'review' would be a stretch of the definiton.
>>>
>>
>> I wasn't aware that PMM wanted to avoid non-standard tags; I consider
>> them to be for human use, but I can change that behavior.
>>
>> Peter, I'll change it to an ACK (as suggested by Kevin) is that's OK by you.
> 
> Sure! I'll spell it out even for compliance:
> 
> ACKed-by: Peter Krempa 
> 

Thanks, applied to my IDE tree:

https://github.com/jnsnow/qemu/commits/ide
https://github.com/jnsnow/qemu.git

--js

Re: [PATCH v2 1/4] qemu-iotests: remove bash shebang from library files

[Kevin Wolf]

Am 11.10.2019 um 22:05 hat Cleber Rosa geschrieben:
> On Fri, Oct 11, 2019 at 02:27:25PM +0300, Nir Soffer wrote:
> > On Fri, Oct 11, 2019, 12:36 Kevin Wolf  wrote:
> > 
> > > Am 09.10.2019 um 21:47 hat Cleber Rosa geschrieben:
> > > > Due to not being able to find a reason to have shebangs on files that
> > > > are not executable.
> > > >
> > > > While at it, add a mode hint to emacs, which would be clueless or
> > > > plain wrong about these containing shell code.
> > >
> > > vim still doesn't like the change.
> > >
> > > Of course, we could also add another line for vim and for every other
> > > editor in use, but actually, I think I'd prefer just dropping this
> > > patch. It even makes each file a few bytes larger instead of saving
> > > something. Shebang lines are a shorter and more portable format
> > > indicator than the alternatives.
> > >
> > > So I think in the end we have found a good reason to keep them. :-)
> > 
> > What about .sh suffix? Should be most portable way.
> 
> That's the approach I tend to follow for my sh code.  Explicit is
> better than implicit if you ask me.

I would certainly agree for new files.

> Kevin,
> 
> Do you have any strong feelings here?  I'd be fine with either this
> or dropping the patch.

No strong feelings. The result of renaming the files would be a bit
nicer than what we have today, but renaming always comes with a cost
when working with the version history later. Hard to tell if it's a net
gain or loss in the end.

Myself, I would probably pick the lazy way and stick with "if it ain't
broke, don't fix it", but I'm not objecting to a change either.

Kevin

Re: [PATCH v2 1/4] qemu-iotests: remove bash shebang from library files

[Cleber Rosa]

On Fri, Oct 11, 2019 at 02:27:25PM +0300, Nir Soffer wrote:
> On Fri, Oct 11, 2019, 12:36 Kevin Wolf  wrote:
> 
> > Am 09.10.2019 um 21:47 hat Cleber Rosa geschrieben:
> > > Due to not being able to find a reason to have shebangs on files that
> > > are not executable.
> > >
> > > While at it, add a mode hint to emacs, which would be clueless or
> > > plain wrong about these containing shell code.
> >
> > vim still doesn't like the change.
> >
> > Of course, we could also add another line for vim and for every other
> > editor in use, but actually, I think I'd prefer just dropping this
> > patch. It even makes each file a few bytes larger instead of saving
> > something. Shebang lines are a shorter and more portable format
> > indicator than the alternatives.
> >
> > So I think in the end we have found a good reason to keep them. :-)
> >
> 
> What about .sh suffix? Should be most portable way.
> 
> >

That's the approach I tend to follow for my sh code.  Explicit is
better than implicit if you ask me.

Kevin,

Do you have any strong feelings here?  I'd be fine with either this
or dropping the patch.

Thanks,
- Cleber.

Re: [RFC v5 099/126] nbd: introduce ERRP_AUTO_PROPAGATE

[Eric Blake]

On 10/11/19 11:05 AM, Vladimir Sementsov-Ogievskiy wrote:

If we want to add some info to errp (by error_prepend() or
error_append_hint()), we must use the ERRP_AUTO_PROPAGATE macro.
Otherwise, this info will not be added when errp == _err
(the program will exit prior to the error_append_hint() or
error_prepend() call).  Fix such cases.

If we want to check error after errp-function call, we need to
introduce local_err and than propagate it to errp. Instead, use
ERRP_AUTO_PROPAGATE macro, benefits are:
1. No need of explicit error_propagate call
2. No need of explicit local_err variable: use errp directly
3. ERRP_AUTO_PROPAGATE leaves errp as is if it's not NULL or
_fatel, this means that we don't break error_abort
(we'll abort on error_set, not on error_propagate)




Reported-by: Kevin Wolf 
Reported-by: Greg Kurz 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 include/block/nbd.h |  1 +
 block/nbd.c | 49 +
 nbd/client.c|  5 +
 nbd/server.c|  5 +
 4 files changed, 34 insertions(+), 26 deletions(-)


One of the few patches with a net gain rather than loss of lines, but 
that's because of lots of error_prepend use (where you add the macro 
without removing error_propagate).


Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

[PULL 19/21] migration/multifd: initialize packet->magic/version once at setup stage

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

MultiFDPacket_t's magic and version field never changes during
migration, so move these two fields in setup stage.

Signed-off-by: Wei Yang 
Message-Id: <20191011085050.17622-4-richardw.y...@linux.intel.com>
Reviewed-by: Juan Quintela 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/ram.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index 84c5953a84..963e795ed0 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -793,8 +793,6 @@ static void multifd_send_fill_packet(MultiFDSendParams *p)
 MultiFDPacket_t *packet = p->packet;
 int i;
 
-packet->magic = cpu_to_be32(MULTIFD_MAGIC);
-packet->version = cpu_to_be32(MULTIFD_VERSION);
 packet->flags = cpu_to_be32(p->flags);
 packet->pages_alloc = cpu_to_be32(p->pages->allocated);
 packet->pages_used = cpu_to_be32(p->pages->used);
@@ -1240,6 +1238,8 @@ int multifd_save_setup(void)
 p->packet_len = sizeof(MultiFDPacket_t)
   + sizeof(ram_addr_t) * page_count;
 p->packet = g_malloc0(p->packet_len);
+p->packet->magic = cpu_to_be32(MULTIFD_MAGIC);
+p->packet->version = cpu_to_be32(MULTIFD_VERSION);
 p->name = g_strdup_printf("multifdsend_%d", i);
 socket_send_channel_create(multifd_new_send_channel_async, p);
 }
-- 
2.23.0

[PULL 16/21] migration/postcopy: check PostcopyState before setting to POSTCOPY_INCOMING_RUNNING

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

Currently, we set PostcopyState blindly to RUNNING, even we found the
previous state is not LISTENING. This will lead to a corner case.

First let's look at the code flow:

qemu_loadvm_state_main()
ret = loadvm_process_command()
loadvm_postcopy_handle_run()
return -1;
if (ret < 0) {
if (postcopy_state_get() == POSTCOPY_INCOMING_RUNNING)
...
}

>From above snippet, the corner case is loadvm_postcopy_handle_run()
always sets state to RUNNING. And then it checks the previous state. If
the previous state is not LISTENING, it will return -1. But at this
moment, PostcopyState is already been set to RUNNING.

Then ret is checked in qemu_loadvm_state_main(), when it is -1
PostcopyState is checked. Current logic would pause postcopy and retry
if PostcopyState is RUNNING. This is not what we expect, because
postcopy is not active yet.

This patch makes sure state is set to RUNNING only previous state is
LISTENING by checking the state first.

Signed-off-by: Wei Yang 
Suggested by: Peter Xu 
Message-Id: <20191010011316.31363-3-richardw.y...@linux.intel.com>
Reviewed-by: Peter Xu 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/savevm.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/migration/savevm.c b/migration/savevm.c
index 95574c4e9d..8d95e261f6 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1933,7 +1933,7 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)
 /* After all discards we can start running and asking for pages */
 static int loadvm_postcopy_handle_run(MigrationIncomingState *mis)
 {
-PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_RUNNING);
+PostcopyState ps = postcopy_state_get();
 
 trace_loadvm_postcopy_handle_run();
 if (ps != POSTCOPY_INCOMING_LISTENING) {
@@ -1941,6 +1941,7 @@ static int 
loadvm_postcopy_handle_run(MigrationIncomingState *mis)
 return -1;
 }
 
+postcopy_state_set(POSTCOPY_INCOMING_RUNNING);
 mis->bh = qemu_bh_new(loadvm_postcopy_handle_run_bh, mis);
 qemu_bh_schedule(mis->bh);
 
-- 
2.23.0

[PULL 14/21] migration/postcopy: postpone setting PostcopyState to END

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

There are two places to call function postcopy_ram_incoming_cleanup()

postcopy_ram_listen_thread on migration success
loadvm_postcopy_handle_listen one setup failure

On success, the vm will never accept another migration. On failure,
PostcopyState is transited from LISTENING to END and would be checked in
qemu_loadvm_state_main(). If PostcopyState is RUNNING, migration would
be paused and retried.

Currently PostcopyState is set to END in function
postcopy_ram_incoming_cleanup(). With above analysis, we can take this
step out and postpone this till the end of listen thread to indicate the
listen thread is done.

This is a preparation patch for later cleanup.

Signed-off-by: Wei Yang 
Message-Id: <20191006000249.29926-3-richardw.y...@linux.intel.com>
Reviewed-by: Dr. David Alan Gilbert 
Signed-off-by: Dr. David Alan Gilbert 
  Fixed up in merge to the 1 parameter postcopy_state_set
---
 migration/postcopy-ram.c | 2 --
 migration/savevm.c   | 2 ++
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 3a72f7b4fe..a793bad477 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -577,8 +577,6 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState 
*mis)
 }
 }
 
-postcopy_state_set(POSTCOPY_INCOMING_END);
-
 if (mis->postcopy_tmp_page) {
 munmap(mis->postcopy_tmp_page, mis->largest_page_size);
 mis->postcopy_tmp_page = NULL;
diff --git a/migration/savevm.c b/migration/savevm.c
index c62687afef..bf3da58ecc 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1837,6 +1837,8 @@ static void *postcopy_ram_listen_thread(void *opaque)
 
 rcu_unregister_thread();
 mis->have_listen_thread = false;
+postcopy_state_set(POSTCOPY_INCOMING_END);
+
 return NULL;
 }
 
-- 
2.23.0

[PULL 13/21] migration/postcopy: mis->have_listen_thread check will never be touched

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

If mis->have_listen_thread is true, this means current PostcopyState
must be LISTENING or RUNNING. While the check at the beginning of the
function makes sure the state transaction happens when its previous
PostcopyState is ADVISE or DISCARD.

This means we would never touch this check.

Signed-off-by: Wei Yang 
Message-Id: <20191006000249.29926-2-richardw.y...@linux.intel.com>
Reviewed-by: Dr. David Alan Gilbert 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/savevm.c | 5 -
 1 file changed, 5 deletions(-)

diff --git a/migration/savevm.c b/migration/savevm.c
index 241c5dd097..c62687afef 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1878,11 +1878,6 @@ static int 
loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
 return -1;
 }
 
-if (mis->have_listen_thread) {
-error_report("CMD_POSTCOPY_RAM_LISTEN already has a listen thread");
-return -1;
-}
-
 mis->have_listen_thread = true;
 /* Start up the listening thread and wait for it to signal ready */
 qemu_sem_init(>listen_thread_sem, 0);
-- 
2.23.0

Re: [PATCH v5 00/22] target/arm: Implement ARMv8.5-MemTag, system mode

[no-reply]

Patchew URL: 
https://patchew.org/QEMU/20191011134744.2477-1-richard.hender...@linaro.org/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [PATCH v5 00/22] target/arm: Implement ARMv8.5-MemTag, system mode
Type: series
Message-id: 20191011134744.2477-1-richard.hender...@linaro.org

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
2cb4ac6 target/arm: Add allocation tag storage for system mode
04c91b4 target/arm: Add mmu indexes for tag memory
1e0ff9d target/arm: Create tagged ram when MTE is enabled
b751fc4 target/arm: Cache the Tagged bit for a page in MemTxAttrs
9ac60fd target/arm: Enable MTE
513b429 target/arm: Set PSTATE.TCO on exception entry
b34fae3 target/arm: Implement data cache set allocation tags
e821379 target/arm: Clean address for DC ZVA
aff90e1 target/arm: Implement the access tag cache flushes
0caee2b target/arm: Implement the LDGM and STGM instructions
2b4a576 target/arm: Implement the STGP instruction
8dc4ae2 target/arm: Implement LDG, STG, ST2G instructions
2f4a984 target/arm: Define arm_cpu_do_unaligned_access for CONFIG_USER_ONLY
3af0a57 target/arm: Implement the SUBP instruction
b26b9b0 target/arm: Implement the GMI instruction
83744f3 target/arm: Implement ADDG, SUBG instructions
454811d target/arm: Implement the IRG instruction
3570a15 target/arm: Suppress tag check for sp+offset
1409fa4 target/arm: Add helper_mte_check{1,2,3}
bbbd12d target/arm: Add MTE system registers
bac1b74 target/arm: Add regime_has_2_ranges
498eda0 target/arm: Add MTE_ACTIVE to tb_flags

=== OUTPUT BEGIN ===
1/22 Checking commit 498eda06038b (target/arm: Add MTE_ACTIVE to tb_flags)
ERROR: spaces prohibited around that ':' (ctx:WxW)
#214: FILE: target/arm/internals.h:986:
+bool tcma   : 1;
 ^

total: 1 errors, 0 warnings, 213 lines checked

Patch 1/22 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

2/22 Checking commit bac1b74f8ee9 (target/arm: Add regime_has_2_ranges)
3/22 Checking commit bbbd12d06479 (target/arm: Add MTE system registers)
4/22 Checking commit 1409fa4954ca (target/arm: Add helper_mte_check{1,2,3})
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#39: 
new file mode 100644

total: 0 errors, 1 warnings, 199 lines checked

Patch 4/22 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
5/22 Checking commit 3570a158fe8a (target/arm: Suppress tag check for sp+offset)
6/22 Checking commit 454811d5fbee (target/arm: Implement the IRG instruction)
7/22 Checking commit 83744f3effbf (target/arm: Implement ADDG, SUBG 
instructions)
8/22 Checking commit b26b9b0ff90d (target/arm: Implement the GMI instruction)
9/22 Checking commit 3af0a5757328 (target/arm: Implement the SUBP instruction)
10/22 Checking commit 2f4a98446331 (target/arm: Define 
arm_cpu_do_unaligned_access for CONFIG_USER_ONLY)
11/22 Checking commit 8dc4ae2e080d (target/arm: Implement LDG, STG, ST2G 
instructions)
12/22 Checking commit 2b4a576928a6 (target/arm: Implement the STGP instruction)
13/22 Checking commit 0caee2b19728 (target/arm: Implement the LDGM and STGM 
instructions)
14/22 Checking commit aff90e1ac887 (target/arm: Implement the access tag cache 
flushes)
15/22 Checking commit e82137999567 (target/arm: Clean address for DC ZVA)
16/22 Checking commit b34fae36a523 (target/arm: Implement data cache set 
allocation tags)
17/22 Checking commit 513b42914b4f (target/arm: Set PSTATE.TCO on exception 
entry)
18/22 Checking commit 9ac60fd7222d (target/arm: Enable MTE)
19/22 Checking commit b751fc4415fe (target/arm: Cache the Tagged bit for a page 
in MemTxAttrs)
20/22 Checking commit 1e0ff9daa24d (target/arm: Create tagged ram when MTE is 
enabled)
21/22 Checking commit 04c91b4cfe18 (target/arm: Add mmu indexes for tag memory)
22/22 Checking commit 2cb4ac638a77 (target/arm: Add allocation tag storage for 
system mode)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20191011134744.2477-1-richard.hender...@linaro.org/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-de...@redhat.com

Re: [RFC v5 080/126] QAPI: introduce ERRP_AUTO_PROPAGATE

[Eric Blake]

On 10/11/19 11:05 AM, Vladimir Sementsov-Ogievskiy wrote:

If we want to add some info to errp (by error_prepend() or
error_append_hint()), we must use the ERRP_AUTO_PROPAGATE macro.
Otherwise, this info will not be added when errp == _err
(the program will exit prior to the error_append_hint() or
error_prepend() call).  Fix such cases.

If we want to check error after errp-function call, we need to
introduce local_err and than propagate it to errp. Instead, use
ERRP_AUTO_PROPAGATE macro, benefits are:
1. No need of explicit error_propagate call
2. No need of explicit local_err variable: use errp directly
3. ERRP_AUTO_PROPAGATE leaves errp as is if it's not NULL or
_fatel, this means that we don't break error_abort
(we'll abort on error_set, not on error_propagate)




Reported-by: Kevin Wolf 
Reported-by: Greg Kurz 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
  qapi/qapi-visit-core.c  | 56 -
  qapi/qmp-dispatch.c |  7 ++---
  qapi/string-input-visitor.c |  7 ++---
  3 files changed, 30 insertions(+), 40 deletions(-)




+++ b/qapi/qmp-dispatch.c
@@ -78,7 +78,7 @@ static QDict *qmp_dispatch_check_obj(const QObject *request, 
bool allow_oob,
  static QObject *do_qmp_dispatch(QmpCommandList *cmds, QObject *request,
  bool allow_oob, Error **errp)
  {
-Error *local_err = NULL;
+ERRP_AUTO_PROPAGATE();
  bool oob;
  const char *command;
  QDict *args, *dict;
@@ -129,9 +129,8 @@ static QObject *do_qmp_dispatch(QmpCommandList *cmds, 
QObject *request,
  qobject_ref(args);
  }
  
-cmd->fn(args, , _err);

-if (local_err) {
-error_propagate(errp, local_err);
+cmd->fn(args, , errp);
+if (*errp) {
  } else if (cmd->options & QCO_NO_SUCCESS_RESP) {


Looks a bit funny with the empty if clause, but not worth changing.


  g_assert(!ret);
  } else if (!ret) {


Reviewed-by: Eric Blake 

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [RFC v5 083/126] QMP: introduce ERRP_AUTO_PROPAGATE

[Eric Blake]

On 10/11/19 11:05 AM, Vladimir Sementsov-Ogievskiy wrote:

If we want to add some info to errp (by error_prepend() or
error_append_hint()), we must use the ERRP_AUTO_PROPAGATE macro.
Otherwise, this info will not be added when errp == _err
(the program will exit prior to the error_append_hint() or
error_prepend() call).  Fix such cases.

If we want to check error after errp-function call, we need to
introduce local_err and than propagate it to errp. Instead, use
ERRP_AUTO_PROPAGATE macro, benefits are:
1. No need of explicit error_propagate call
2. No need of explicit local_err variable: use errp directly
3. ERRP_AUTO_PROPAGATE leaves errp as is if it's not NULL or
_fatel, this means that we don't break error_abort
(we'll abort on error_set, not on error_propagate)




Reported-by: Kevin Wolf 
Reported-by: Greg Kurz 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
  monitor/qmp-cmds.c | 7 +++
  1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index c6faa3eaf0..2db3199a2e 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -126,9 +126,9 @@ void qmp_x_exit_preconfig(Error **errp)
  
  void qmp_cont(Error **errp)

  {
+ERRP_AUTO_PROPAGATE();
  BlockBackend *blk;
  BlockJob *job;
-Error *local_err = NULL;
  
  /* if there is a dump in background, we should wait until the dump

   * finished */
@@ -161,9 +161,8 @@ void qmp_cont(Error **errp)
   * If there are no inactive block nodes (e.g. because the VM was just
   * paused rather than completing a migration), bdrv_inactivate_all() 
simply
   * doesn't do anything. */
-bdrv_invalidate_cache_all(_err);
-if (local_err) {
-error_propagate(errp, local_err);
+bdrv_invalidate_cache_all(errp);
+if (*errp) {
  return;
  }
  


Reviewed-by: Eric Blake 

[As elsewhere in this RFC series, my first pass on review is limited to 
just checking that what Coccinelle did is correct, rather than also 
checking whether Coccinelle missed things or whether I could reproduce 
the Coccinelle results]


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

[PULL 10/21] migration/postcopy: fix typo in mark_postcopy_blocktime_begin's comment

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

Signed-off-by: Wei Yang 
Message-Id: <20191005220517.24029-3-richardw.y...@linux.intel.com>
Reviewed-by: Philippe Mathieu-Daudé 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/postcopy-ram.c | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 08a3ed516e..3a72f7b4fe 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -768,9 +768,11 @@ static void mark_postcopy_blocktime_begin(uintptr_t addr, 
uint32_t ptid,
 atomic_xchg(>page_fault_vcpu_time[cpu], low_time_offset);
 atomic_xchg(>vcpu_addr[cpu], addr);
 
-/* check it here, not at the begining of the function,
- * due to, check could accur early than bitmap_set in
- * qemu_ufd_copy_ioctl */
+/*
+ * check it here, not at the beginning of the function,
+ * due to, check could occur early than bitmap_set in
+ * qemu_ufd_copy_ioctl
+ */
 already_received = ramblock_recv_bitmap_test(rb, (void *)addr);
 if (already_received) {
 atomic_xchg(>vcpu_addr[cpu], 0);
-- 
2.23.0

Re: ACPI table modifications

[Philippe Mathieu-Daudé]

Hi Gautam,

On 10/11/19 8:47 PM, Gautam Bhat wrote:

Hi,

I want to add some I2C based temperature sensors to the -M Q35 machine. 
I want to update the ACPI tables to add this device information. How can 
I go about  doing this?


QEMU provides a MAINTAINERS file with persons to contact sorted by 
topics (I Cc'ed in this reply).


This is the relevant ACPI section, it also lists the files you might 
look at:


ACPI/SMBIOS
M: Michael S. Tsirkin
M: Igor Mammedov
S: Supported
F: include/hw/acpi/*
F: include/hw/firmware/smbios.h
F: hw/mem/*
F: hw/acpi/*
F: hw/smbios/*
F: hw/i386/acpi-build.[hc]
F: hw/arm/virt-acpi-build.c
F: tests/bios-tables-test.c
F: tests/acpi-utils.[hc]
F: tests/data/acpi/

Eventually the bios-tables-test.c file is a good starting point.

Regards,

Phil.

[PULL 21/21] migration: Support gtree migration

[Dr. David Alan Gilbert (git)]

From: Eric Auger 

Introduce support for GTree migration. A custom save/restore
is implemented. Each item is made of a key and a data.

If the key is a pointer to an object, 2 VMSDs are passed into
the GTree VMStateField.

When putting the items, the tree is traversed in sorted order by
g_tree_foreach.

On the get() path, gtrees must be allocated using the proper
key compare, key destroy and value destroy. This must be handled
beforehand, for example in a pre_load method.

Tests are added to test save/dump of structs containing gtrees
including the virtio-iommu domain/mappings scenario.

Signed-off-by: Eric Auger 

Message-Id: <20191011121724.433-1-eric.au...@redhat.com>
Reviewed-by: Dr. David Alan Gilbert 
Reviewed-by: Juan Quintela 
Signed-off-by: Dr. David Alan Gilbert 
  uintptr_t fixup for test on 32bit
---
 include/migration/vmstate.h |  40 
 migration/trace-events  |   5 +
 migration/vmstate-types.c   | 152 +
 tests/test-vmstate.c| 421 
 4 files changed, 618 insertions(+)

diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 1fbfd099dd..b9ee563aa4 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -224,6 +224,7 @@ extern const VMStateInfo vmstate_info_unused_buffer;
 extern const VMStateInfo vmstate_info_tmp;
 extern const VMStateInfo vmstate_info_bitmap;
 extern const VMStateInfo vmstate_info_qtailq;
+extern const VMStateInfo vmstate_info_gtree;
 
 #define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)
 /*
@@ -754,6 +755,45 @@ extern const VMStateInfo vmstate_info_qtailq;
 .start= offsetof(_type, _next),  \
 }
 
+/*
+ * For migrating a GTree whose key is a pointer to _key_type and the
+ * value, a pointer to _val_type
+ * The target tree must have been properly initialized
+ * _vmsd: Start address of the 2 element array containing the data vmsd
+ *and the key vmsd, in that order
+ * _key_type: type of the key
+ * _val_type: type of the value
+ */
+#define VMSTATE_GTREE_V(_field, _state, _version, _vmsd,   
\
+_key_type, _val_type)  
\
+{  
\
+.name = (stringify(_field)),   
\
+.version_id   = (_version),
\
+.vmsd = (_vmsd),   
\
+.info = _info_gtree,   
\
+.start= sizeof(_key_type), 
\
+.size = sizeof(_val_type), 
\
+.offset   = offsetof(_state, _field),  
\
+}
+
+/*
+ * For migrating a GTree with direct key and the value a pointer
+ * to _val_type
+ * The target tree must have been properly initialized
+ * _vmsd: data vmsd
+ * _val_type: type of the value
+ */
+#define VMSTATE_GTREE_DIRECT_KEY_V(_field, _state, _version, _vmsd, _val_type) 
\
+{  
\
+.name = (stringify(_field)),   
\
+.version_id   = (_version),
\
+.vmsd = (_vmsd),   
\
+.info = _info_gtree,   
\
+.start= 0, 
\
+.size = sizeof(_val_type), 
\
+.offset   = offsetof(_state, _field),  
\
+}
+
 /* _f : field name
_f_n : num of elements field_name
_n : num of elements
diff --git a/migration/trace-events b/migration/trace-events
index 858d415d56..6dee7b5389 100644
--- a/migration/trace-events
+++ b/migration/trace-events
@@ -71,6 +71,11 @@ get_qtailq_end(const char *name, const char *reason, int 
val) "%s %s/%d"
 put_qtailq(const char *name, int version_id) "%s v%d"
 put_qtailq_end(const char *name, const char *reason) "%s %s"
 
+get_gtree(const char *field_name, const char *key_vmsd_name, const char 
*val_vmsd_name, uint32_t nnodes) "%s(%s/%s) nnodes=%d"
+get_gtree_end(const char *field_name, const char *key_vmsd_name, const char 
*val_vmsd_name, int ret) "%s(%s/%s) %d"
+put_gtree(const char *field_name, const char *key_vmsd_name, const char 
*val_vmsd_name, uint32_t nnodes) "%s(%s/%s) nnodes=%d"
+put_gtree_end(const char *field_name, const char *key_vmsd_name, const char 
*val_vmsd_name, int ret) "%s(%s/%s) %d"
+
 # qemu-file.c
 qemu_file_fclose(void) ""
 
diff --git a/migration/vmstate-types.c b/migration/vmstate-types.c
index bee658a1b2..7236cf92bc 100644
--- a/migration/vmstate-types.c
+++ b/migration/vmstate-types.c
@@ -691,3 

[PULL 06/21] rcu: Use automatic rc_read unlock in core memory/exec code

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" 

Signed-off-by: Dr. David Alan Gilbert 
Reviewed-by: Daniel P. Berrangé 
Message-Id: <20191007143642.301445-6-dgilb...@redhat.com>
Signed-off-by: Dr. David Alan Gilbert 
---
 exec.c  | 116 ++---
 include/exec/ram_addr.h | 138 +++-
 memory.c|  15 ++---
 3 files changed, 118 insertions(+), 151 deletions(-)

diff --git a/exec.c b/exec.c
index bdcfcdff3f..fb0943cfed 100644
--- a/exec.c
+++ b/exec.c
@@ -1037,16 +1037,14 @@ void tb_invalidate_phys_addr(AddressSpace *as, hwaddr 
addr, MemTxAttrs attrs)
 return;
 }
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 mr = address_space_translate(as, addr, , , false, attrs);
 if (!(memory_region_is_ram(mr)
   || memory_region_is_romd(mr))) {
-rcu_read_unlock();
 return;
 }
 ram_addr = memory_region_get_ram_addr(mr) + addr;
 tb_invalidate_phys_page_range(ram_addr, ram_addr + 1);
-rcu_read_unlock();
 }
 
 static void breakpoint_invalidate(CPUState *cpu, target_ulong pc)
@@ -1332,14 +1330,13 @@ static void tlb_reset_dirty_range_all(ram_addr_t start, 
ram_addr_t length)
 end = TARGET_PAGE_ALIGN(start + length);
 start &= TARGET_PAGE_MASK;
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 block = qemu_get_ram_block(start);
 assert(block == qemu_get_ram_block(end - 1));
 start1 = (uintptr_t)ramblock_ptr(block, start - block->offset);
 CPU_FOREACH(cpu) {
 tlb_reset_dirty(cpu, start1, length);
 }
-rcu_read_unlock();
 }
 
 /* Note: start and end must be within the same ram block.  */
@@ -1360,30 +1357,29 @@ bool 
cpu_physical_memory_test_and_clear_dirty(ram_addr_t start,
 end = TARGET_PAGE_ALIGN(start + length) >> TARGET_PAGE_BITS;
 page = start >> TARGET_PAGE_BITS;
 
-rcu_read_lock();
+WITH_RCU_READ_LOCK_GUARD() {
+blocks = atomic_rcu_read(_list.dirty_memory[client]);
+ramblock = qemu_get_ram_block(start);
+/* Range sanity check on the ramblock */
+assert(start >= ramblock->offset &&
+   start + length <= ramblock->offset + ramblock->used_length);
 
-blocks = atomic_rcu_read(_list.dirty_memory[client]);
-ramblock = qemu_get_ram_block(start);
-/* Range sanity check on the ramblock */
-assert(start >= ramblock->offset &&
-   start + length <= ramblock->offset + ramblock->used_length);
+while (page < end) {
+unsigned long idx = page / DIRTY_MEMORY_BLOCK_SIZE;
+unsigned long offset = page % DIRTY_MEMORY_BLOCK_SIZE;
+unsigned long num = MIN(end - page,
+DIRTY_MEMORY_BLOCK_SIZE - offset);
 
-while (page < end) {
-unsigned long idx = page / DIRTY_MEMORY_BLOCK_SIZE;
-unsigned long offset = page % DIRTY_MEMORY_BLOCK_SIZE;
-unsigned long num = MIN(end - page, DIRTY_MEMORY_BLOCK_SIZE - offset);
+dirty |= bitmap_test_and_clear_atomic(blocks->blocks[idx],
+  offset, num);
+page += num;
+}
 
-dirty |= bitmap_test_and_clear_atomic(blocks->blocks[idx],
-  offset, num);
-page += num;
+mr_offset = (ram_addr_t)(page << TARGET_PAGE_BITS) - ramblock->offset;
+mr_size = (end - page) << TARGET_PAGE_BITS;
+memory_region_clear_dirty_bitmap(ramblock->mr, mr_offset, mr_size);
 }
 
-mr_offset = (ram_addr_t)(page << TARGET_PAGE_BITS) - ramblock->offset;
-mr_size = (end - page) << TARGET_PAGE_BITS;
-memory_region_clear_dirty_bitmap(ramblock->mr, mr_offset, mr_size);
-
-rcu_read_unlock();
-
 if (dirty && tcg_enabled()) {
 tlb_reset_dirty_range_all(start, length);
 }
@@ -1411,28 +1407,27 @@ DirtyBitmapSnapshot 
*cpu_physical_memory_snapshot_and_clear_dirty
 end  = last  >> TARGET_PAGE_BITS;
 dest = 0;
 
-rcu_read_lock();
+WITH_RCU_READ_LOCK_GUARD() {
+blocks = atomic_rcu_read(_list.dirty_memory[client]);
 
-blocks = atomic_rcu_read(_list.dirty_memory[client]);
+while (page < end) {
+unsigned long idx = page / DIRTY_MEMORY_BLOCK_SIZE;
+unsigned long offset = page % DIRTY_MEMORY_BLOCK_SIZE;
+unsigned long num = MIN(end - page,
+DIRTY_MEMORY_BLOCK_SIZE - offset);
 
-while (page < end) {
-unsigned long idx = page / DIRTY_MEMORY_BLOCK_SIZE;
-unsigned long offset = page % DIRTY_MEMORY_BLOCK_SIZE;
-unsigned long num = MIN(end - page, DIRTY_MEMORY_BLOCK_SIZE - offset);
-
-assert(QEMU_IS_ALIGNED(offset, (1 << BITS_PER_LEVEL)));
-assert(QEMU_IS_ALIGNED(num,(1 << BITS_PER_LEVEL)));
-offset >>= BITS_PER_LEVEL;
+assert(QEMU_IS_ALIGNED(offset, (1 << BITS_PER_LEVEL)));
+assert(QEMU_IS_ALIGNED(num,(1 << 

[PULL 18/21] migration/multifd: use pages->allocated instead of the static max

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

multifd_send_fill_packet() prepares meta data for following pages to
transfer. It would be more proper to fill pages->allocated instead of
static max value, especially we want to support flexible packet size.

Signed-off-by: Wei Yang 
Message-Id: <20191011085050.17622-3-richardw.y...@linux.intel.com>
Reviewed-by: Juan Quintela 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/ram.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index 698ba4d669..84c5953a84 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -791,13 +791,12 @@ static void multifd_pages_clear(MultiFDPages_t *pages)
 static void multifd_send_fill_packet(MultiFDSendParams *p)
 {
 MultiFDPacket_t *packet = p->packet;
-uint32_t page_max = MULTIFD_PACKET_SIZE / qemu_target_page_size();
 int i;
 
 packet->magic = cpu_to_be32(MULTIFD_MAGIC);
 packet->version = cpu_to_be32(MULTIFD_VERSION);
 packet->flags = cpu_to_be32(p->flags);
-packet->pages_alloc = cpu_to_be32(page_max);
+packet->pages_alloc = cpu_to_be32(p->pages->allocated);
 packet->pages_used = cpu_to_be32(p->pages->used);
 packet->next_packet_size = cpu_to_be32(p->next_packet_size);
 packet->packet_num = cpu_to_be64(p->packet_num);
-- 
2.23.0

[PULL 15/21] migration/postcopy: rename postcopy_ram_enable_notify to postcopy_ram_incoming_setup

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

Function postcopy_ram_incoming_setup and postcopy_ram_incoming_cleanup
is a pair. Rename to make it clear for audience.

Signed-off-by: Wei Yang 
Reviewed-by: Dr. David Alan Gilbert 
Message-Id: <20191010011316.31363-2-richardw.y...@linux.intel.com>
Reviewed-by: Peter Xu 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/postcopy-ram.c | 4 ++--
 migration/postcopy-ram.h | 2 +-
 migration/savevm.c   | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index a793bad477..abccafc8c8 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -1094,7 +1094,7 @@ retry:
 return NULL;
 }
 
-int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
 {
 /* Open the fd for the kernel to give us userfaults */
 mis->userfault_fd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
@@ -1307,7 +1307,7 @@ int postcopy_request_shared_page(struct PostCopyFD *pcfd, 
RAMBlock *rb,
 return -1;
 }
 
-int postcopy_ram_enable_notify(MigrationIncomingState *mis)
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
 {
 assert(0);
 return -1;
diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h
index fb4cd11d28..9941feb63a 100644
--- a/migration/postcopy-ram.h
+++ b/migration/postcopy-ram.h
@@ -20,7 +20,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState 
*mis);
  * Make all of RAM sensitive to accesses to areas that haven't yet been written
  * and wire up anything necessary to deal with it.
  */
-int postcopy_ram_enable_notify(MigrationIncomingState *mis);
+int postcopy_ram_incoming_setup(MigrationIncomingState *mis);
 
 /*
  * Initialise postcopy-ram, setting the RAM to a state where we can go into
diff --git a/migration/savevm.c b/migration/savevm.c
index bf3da58ecc..95574c4e9d 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1869,7 +1869,7 @@ static int 
loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
  * shouldn't be doing anything yet so don't actually expect requests
  */
 if (migrate_postcopy_ram()) {
-if (postcopy_ram_enable_notify(mis)) {
+if (postcopy_ram_incoming_setup(mis)) {
 postcopy_ram_incoming_cleanup(mis);
 return -1;
 }
-- 
2.23.0

[PULL 04/21] migration: Use automatic rcu_read unlock in ram.c

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" 

Use the automatic read unlocker in migration/ram.c

Signed-off-by: Dr. David Alan Gilbert 
Reviewed-by: Daniel P. Berrangé 
Message-Id: <20191007143642.301445-4-dgilb...@redhat.com>
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/ram.c | 259 ++--
 1 file changed, 121 insertions(+), 138 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index 484d66379a..e29c8b3408 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -181,14 +181,14 @@ int foreach_not_ignored_block(RAMBlockIterFunc func, void 
*opaque)
 RAMBlock *block;
 int ret = 0;
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
+
 RAMBLOCK_FOREACH_NOT_IGNORED(block) {
 ret = func(block, opaque);
 if (ret) {
 break;
 }
 }
-rcu_read_unlock();
 return ret;
 }
 
@@ -1848,12 +1848,12 @@ static void migration_bitmap_sync(RAMState *rs)
 memory_global_dirty_log_sync();
 
 qemu_mutex_lock(>bitmap_mutex);
-rcu_read_lock();
-RAMBLOCK_FOREACH_NOT_IGNORED(block) {
-ramblock_sync_dirty_bitmap(rs, block);
+WITH_RCU_READ_LOCK_GUARD() {
+RAMBLOCK_FOREACH_NOT_IGNORED(block) {
+ramblock_sync_dirty_bitmap(rs, block);
+}
+ram_counters.remaining = ram_bytes_remaining();
 }
-ram_counters.remaining = ram_bytes_remaining();
-rcu_read_unlock();
 qemu_mutex_unlock(>bitmap_mutex);
 
 memory_global_after_dirty_log_sync();
@@ -2397,13 +2397,12 @@ static void migration_page_queue_free(RAMState *rs)
 /* This queue generally should be empty - but in the case of a failed
  * migration might have some droppings in.
  */
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 QSIMPLEQ_FOREACH_SAFE(mspr, >src_page_requests, next_req, next_mspr) {
 memory_region_unref(mspr->rb->mr);
 QSIMPLEQ_REMOVE_HEAD(>src_page_requests, next_req);
 g_free(mspr);
 }
-rcu_read_unlock();
 }
 
 /**
@@ -2424,7 +2423,8 @@ int ram_save_queue_pages(const char *rbname, ram_addr_t 
start, ram_addr_t len)
 RAMState *rs = ram_state;
 
 ram_counters.postcopy_requests++;
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
+
 if (!rbname) {
 /* Reuse last RAMBlock */
 ramblock = rs->last_req_rb;
@@ -2466,12 +2466,10 @@ int ram_save_queue_pages(const char *rbname, ram_addr_t 
start, ram_addr_t len)
 QSIMPLEQ_INSERT_TAIL(>src_page_requests, new_entry, next_req);
 migration_make_urgent_request();
 qemu_mutex_unlock(>src_page_req_mutex);
-rcu_read_unlock();
 
 return 0;
 
 err:
-rcu_read_unlock();
 return -1;
 }
 
@@ -2700,7 +2698,8 @@ static uint64_t ram_bytes_total_common(bool count_ignored)
 RAMBlock *block;
 uint64_t total = 0;
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
+
 if (count_ignored) {
 RAMBLOCK_FOREACH_MIGRATABLE(block) {
 total += block->used_length;
@@ -2710,7 +2709,6 @@ static uint64_t ram_bytes_total_common(bool count_ignored)
 total += block->used_length;
 }
 }
-rcu_read_unlock();
 return total;
 }
 
@@ -3034,7 +3032,7 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms)
 RAMBlock *block;
 int ret;
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 
 /* This should be our last sync, the src is now paused */
 migration_bitmap_sync(rs);
@@ -3048,7 +3046,6 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms)
 /* Deal with TPS != HPS and huge pages */
 ret = postcopy_chunk_hostpages(ms, block);
 if (ret) {
-rcu_read_unlock();
 return ret;
 }
 
@@ -3060,7 +3057,6 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms)
 trace_ram_postcopy_send_discard_bitmap();
 
 ret = postcopy_each_ram_send_discard(ms);
-rcu_read_unlock();
 
 return ret;
 }
@@ -3081,7 +3077,7 @@ int ram_discard_range(const char *rbname, uint64_t start, 
size_t length)
 
 trace_ram_discard_range(rbname, start, length);
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 RAMBlock *rb = qemu_ram_block_by_name(rbname);
 
 if (!rb) {
@@ -3101,8 +3097,6 @@ int ram_discard_range(const char *rbname, uint64_t start, 
size_t length)
 ret = ram_block_discard_range(rb, start, length);
 
 err:
-rcu_read_unlock();
-
 return ret;
 }
 
@@ -3231,13 +3225,12 @@ static void ram_init_bitmaps(RAMState *rs)
 /* For memory_global_dirty_log_start below.  */
 qemu_mutex_lock_iothread();
 qemu_mutex_lock_ramlist();
-rcu_read_lock();
 
-ram_list_init_bitmaps();
-memory_global_dirty_log_start();
-migration_bitmap_sync_precopy(rs);
-
-rcu_read_unlock();
+WITH_RCU_READ_LOCK_GUARD() {
+ram_list_init_bitmaps();
+memory_global_dirty_log_start();
+migration_bitmap_sync_precopy(rs);
+}
 qemu_mutex_unlock_ramlist();
 qemu_mutex_unlock_iothread();
 }
@@ -3424,55 +3417,57 

[PULL 17/21] migration/multifd: fix a typo in comment of multifd_recv_unfill_packet()

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

Signed-off-by: Wei Yang 
Message-Id: <20191011085050.17622-2-richardw.y...@linux.intel.com>
Reviewed-by: Juan Quintela 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/ram.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/migration/ram.c b/migration/ram.c
index 071687ef37..698ba4d669 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -838,7 +838,7 @@ static int multifd_recv_unfill_packet(MultiFDRecvParams *p, 
Error **errp)
 
 packet->pages_alloc = be32_to_cpu(packet->pages_alloc);
 /*
- * If we recevied a packet that is 100 times bigger than expected
+ * If we received a packet that is 100 times bigger than expected
  * just stop migration.  It is a magic number.
  */
 if (packet->pages_alloc > pages_max * 100) {
-- 
2.23.0

[PULL 20/21] migration/multifd: pages->used would be cleared when attach to multifd_send_state

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

When we found an available channel in multifd_send_pages(), its
pages->used is cleared and then attached to multifd_send_state.

It is not necessary to do this twice.

Signed-off-by: Wei Yang 
Message-Id: <20191011085050.17622-5-richardw.y...@linux.intel.com>
Reviewed-by: Juan Quintela 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/ram.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/migration/ram.c b/migration/ram.c
index 963e795ed0..5078f94490 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -1129,7 +1129,6 @@ static void *multifd_send_thread(void *opaque)
 p->flags = 0;
 p->num_packets++;
 p->num_pages += used;
-p->pages->used = 0;
 qemu_mutex_unlock(>mutex);
 
 trace_multifd_send(p->id, packet_num, used, flags,
-- 
2.23.0

[PULL 12/21] migration: report SaveStateEntry id and name on failure

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

This provides helpful information on which entry failed.

Signed-off-by: Wei Yang 
Message-Id: <20191005220517.24029-5-richardw.y...@linux.intel.com>
Reviewed-by: Philippe Mathieu-Daudé 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/savevm.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/migration/savevm.c b/migration/savevm.c
index bb9462a54d..241c5dd097 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1215,6 +1215,8 @@ int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy)
 save_section_footer(f, se);
 
 if (ret < 0) {
+error_report("failed to save SaveStateEntry with id(name): %d(%s)",
+ se->section_id, se->idstr);
 qemu_file_set_error(f, ret);
 }
 if (ret <= 0) {
-- 
2.23.0

[PULL 03/21] migration: Fix missing rcu_read_unlock

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" 

Use the automatic rcu_read unlocker to fix a missing unlock.

Signed-off-by: Dr. David Alan Gilbert 
Reviewed-by: Daniel P. Berrangé 
Message-Id: <20191007143642.301445-3-dgilb...@redhat.com>
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/ram.c | 27 +--
 1 file changed, 13 insertions(+), 14 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index 22423f08cd..484d66379a 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -3373,24 +3373,23 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
 }
 (*rsp)->f = f;
 
-rcu_read_lock();
-
-qemu_put_be64(f, ram_bytes_total_common(true) | RAM_SAVE_FLAG_MEM_SIZE);
+WITH_RCU_READ_LOCK_GUARD() {
+qemu_put_be64(f, ram_bytes_total_common(true) | 
RAM_SAVE_FLAG_MEM_SIZE);
 
-RAMBLOCK_FOREACH_MIGRATABLE(block) {
-qemu_put_byte(f, strlen(block->idstr));
-qemu_put_buffer(f, (uint8_t *)block->idstr, strlen(block->idstr));
-qemu_put_be64(f, block->used_length);
-if (migrate_postcopy_ram() && block->page_size != qemu_host_page_size) 
{
-qemu_put_be64(f, block->page_size);
-}
-if (migrate_ignore_shared()) {
-qemu_put_be64(f, block->mr->addr);
+RAMBLOCK_FOREACH_MIGRATABLE(block) {
+qemu_put_byte(f, strlen(block->idstr));
+qemu_put_buffer(f, (uint8_t *)block->idstr, strlen(block->idstr));
+qemu_put_be64(f, block->used_length);
+if (migrate_postcopy_ram() && block->page_size !=
+  qemu_host_page_size) {
+qemu_put_be64(f, block->page_size);
+}
+if (migrate_ignore_shared()) {
+qemu_put_be64(f, block->mr->addr);
+}
 }
 }
 
-rcu_read_unlock();
-
 ram_control_before_iterate(f, RAM_CONTROL_SETUP);
 ram_control_after_iterate(f, RAM_CONTROL_SETUP);
 
-- 
2.23.0

[PULL 11/21] migration: pass in_postcopy instead of check state again

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

Not necessary to do the check again.

Signed-off-by: Wei Yang 
Message-Id: <20191005220517.24029-4-richardw.y...@linux.intel.com>
Reviewed-by: Dr. David Alan Gilbert 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/migration.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index d7f8b428e0..3febd0f8f3 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -3149,8 +3149,7 @@ static MigIterateState 
migration_iteration_run(MigrationState *s)
 return MIG_ITERATE_SKIP;
 }
 /* Just another iteration step */
-qemu_savevm_state_iterate(s->to_dst_file,
-s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE);
+qemu_savevm_state_iterate(s->to_dst_file, in_postcopy);
 } else {
 trace_migration_thread_low_pending(pending_size);
 migration_completion(s);
-- 
2.23.0

[PULL 09/21] migration/postcopy: map large zero page in postcopy_ram_incoming_setup()

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

postcopy_ram_incoming_setup() and postcopy_ram_incoming_cleanup() are
counterpart. It is reasonable to map/unmap large zero page in these two
functions respectively.

Signed-off-by: Wei Yang 
Message-Id: <20191005135021.21721-3-richardw.y...@linux.intel.com>
Reviewed-by: Dr. David Alan Gilbert 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/postcopy-ram.c | 34 +-
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 9a16f52a79..08a3ed516e 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -1144,6 +1144,22 @@ int postcopy_ram_enable_notify(MigrationIncomingState 
*mis)
 return -1;
 }
 
+/*
+ * Map large zero page when kernel can't use UFFDIO_ZEROPAGE for hugepages
+ */
+mis->postcopy_tmp_zero_page = mmap(NULL, mis->largest_page_size,
+   PROT_READ | PROT_WRITE,
+   MAP_PRIVATE | MAP_ANONYMOUS,
+   -1, 0);
+if (mis->postcopy_tmp_zero_page == MAP_FAILED) {
+int e = errno;
+mis->postcopy_tmp_zero_page = NULL;
+error_report("%s: Failed to map large zero page %s",
+ __func__, strerror(e));
+return -e;
+}
+memset(mis->postcopy_tmp_zero_page, '\0', mis->largest_page_size);
+
 /*
  * Ballooning can mark pages as absent while we're postcopying
  * that would cause false userfaults.
@@ -1250,23 +1266,7 @@ int postcopy_place_page_zero(MigrationIncomingState 
*mis, void *host,
qemu_ram_block_host_offset(rb,
   host));
 } else {
-/* The kernel can't use UFFDIO_ZEROPAGE for hugepages */
-if (!mis->postcopy_tmp_zero_page) {
-mis->postcopy_tmp_zero_page = mmap(NULL, mis->largest_page_size,
-   PROT_READ | PROT_WRITE,
-   MAP_PRIVATE | MAP_ANONYMOUS,
-   -1, 0);
-if (mis->postcopy_tmp_zero_page == MAP_FAILED) {
-int e = errno;
-mis->postcopy_tmp_zero_page = NULL;
-error_report("%s: %s mapping large zero page",
- __func__, strerror(e));
-return -e;
-}
-memset(mis->postcopy_tmp_zero_page, '\0', mis->largest_page_size);
-}
-return postcopy_place_page(mis, host, mis->postcopy_tmp_zero_page,
-   rb);
+return postcopy_place_page(mis, host, mis->postcopy_tmp_zero_page, rb);
 }
 }
 
-- 
2.23.0

[PULL 05/21] migration: Use automatic rcu_read unlock in rdma.c

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" 

Use the automatic read unlocker in migration/rdma.c.

Signed-off-by: Dr. David Alan Gilbert 
Reviewed-by: Daniel P. Berrangé 
Message-Id: <20191007143642.301445-5-dgilb...@redhat.com>
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/rdma.c | 57 ++--
 1 file changed, 11 insertions(+), 46 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 4c74e88a37..e241dcb992 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -88,7 +88,6 @@ static uint32_t known_capabilities = RDMA_CAPABILITY_PIN_ALL;
 " to abort!"); \
 rdma->error_reported = 1; \
 } \
-rcu_read_unlock(); \
 return rdma->error_state; \
 } \
 } while (0)
@@ -2678,11 +2677,10 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc,
 size_t i;
 size_t len = 0;
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 rdma = atomic_rcu_read(>rdmaout);
 
 if (!rdma) {
-rcu_read_unlock();
 return -EIO;
 }
 
@@ -2695,7 +2693,6 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc,
 ret = qemu_rdma_write_flush(f, rdma);
 if (ret < 0) {
 rdma->error_state = ret;
-rcu_read_unlock();
 return ret;
 }
 
@@ -2715,7 +2712,6 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc,
 
 if (ret < 0) {
 rdma->error_state = ret;
-rcu_read_unlock();
 return ret;
 }
 
@@ -2724,7 +2720,6 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc,
 }
 }
 
-rcu_read_unlock();
 return done;
 }
 
@@ -2764,11 +2759,10 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc,
 ssize_t i;
 size_t done = 0;
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 rdma = atomic_rcu_read(>rdmain);
 
 if (!rdma) {
-rcu_read_unlock();
 return -EIO;
 }
 
@@ -2805,7 +2799,6 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc,
 
 if (ret < 0) {
 rdma->error_state = ret;
-rcu_read_unlock();
 return ret;
 }
 
@@ -2819,14 +2812,12 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc,
 /* Still didn't get enough, so lets just return */
 if (want) {
 if (done == 0) {
-rcu_read_unlock();
 return QIO_CHANNEL_ERR_BLOCK;
 } else {
 break;
 }
 }
 }
-rcu_read_unlock();
 return done;
 }
 
@@ -2882,7 +2873,7 @@ qio_channel_rdma_source_prepare(GSource *source,
 GIOCondition cond = 0;
 *timeout = -1;
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 if (rsource->condition == G_IO_IN) {
 rdma = atomic_rcu_read(>rioc->rdmain);
 } else {
@@ -2891,7 +2882,6 @@ qio_channel_rdma_source_prepare(GSource *source,
 
 if (!rdma) {
 error_report("RDMAContext is NULL when prepare Gsource");
-rcu_read_unlock();
 return FALSE;
 }
 
@@ -2900,7 +2890,6 @@ qio_channel_rdma_source_prepare(GSource *source,
 }
 cond |= G_IO_OUT;
 
-rcu_read_unlock();
 return cond & rsource->condition;
 }
 
@@ -2911,7 +2900,7 @@ qio_channel_rdma_source_check(GSource *source)
 RDMAContext *rdma;
 GIOCondition cond = 0;
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 if (rsource->condition == G_IO_IN) {
 rdma = atomic_rcu_read(>rioc->rdmain);
 } else {
@@ -2920,7 +2909,6 @@ qio_channel_rdma_source_check(GSource *source)
 
 if (!rdma) {
 error_report("RDMAContext is NULL when check Gsource");
-rcu_read_unlock();
 return FALSE;
 }
 
@@ -2929,7 +2917,6 @@ qio_channel_rdma_source_check(GSource *source)
 }
 cond |= G_IO_OUT;
 
-rcu_read_unlock();
 return cond & rsource->condition;
 }
 
@@ -2943,7 +2930,7 @@ qio_channel_rdma_source_dispatch(GSource *source,
 RDMAContext *rdma;
 GIOCondition cond = 0;
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 if (rsource->condition == G_IO_IN) {
 rdma = atomic_rcu_read(>rioc->rdmain);
 } else {
@@ -2952,7 +2939,6 @@ qio_channel_rdma_source_dispatch(GSource *source,
 
 if (!rdma) {
 error_report("RDMAContext is NULL when dispatch Gsource");
-rcu_read_unlock();
 return FALSE;
 }
 
@@ -2961,7 +2947,6 @@ qio_channel_rdma_source_dispatch(GSource *source,
 }
 cond |= G_IO_OUT;
 
-rcu_read_unlock();
 return (*func)(QIO_CHANNEL(rsource->rioc),
(cond & rsource->condition),
user_data);
@@ -3073,7 +3058,7 @@ qio_channel_rdma_shutdown(QIOChannel *ioc,
 QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
 RDMAContext *rdmain, *rdmaout;
 
-rcu_read_lock();
+RCU_READ_LOCK_GUARD();
 
 rdmain = atomic_rcu_read(>rdmain);
 rdmaout = atomic_rcu_read(>rdmain);
@@ -3100,7 +3085,6 @@ 

[PULL 02/21] rcu: Add automatically released rcu_read_lock variants

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" 

RCU_READ_LOCK_GUARD() takes the rcu_read_lock and then uses glib's
g_auto infrastructure (and thus whatever the compiler's hooks are) to
release it on all exits of the block.

WITH_RCU_READ_LOCK_GUARD() is similar but is used as a wrapper for the
lock, i.e.:

   WITH_RCU_READ_LOCK_GUARD() {
   stuff under lock
   }

Note the 'unused' attribute is needed to work around clang bug:
  https://bugs.llvm.org/show_bug.cgi?id=43482

Signed-off-by: Dr. David Alan Gilbert 
Acked-by: Paolo Bonzini 
Reviewed-by: Daniel P. Berrangé 
Message-Id: <20191007143642.301445-2-dgilb...@redhat.com>
Signed-off-by: Dr. David Alan Gilbert 
---
 docs/devel/rcu.txt | 16 
 include/qemu/rcu.h | 25 +
 2 files changed, 41 insertions(+)

diff --git a/docs/devel/rcu.txt b/docs/devel/rcu.txt
index c84e7f42b2..d83fed2f79 100644
--- a/docs/devel/rcu.txt
+++ b/docs/devel/rcu.txt
@@ -187,6 +187,22 @@ The following APIs must be used before RCU is used in a 
thread:
 Note that these APIs are relatively heavyweight, and should _not_ be
 nested.
 
+Convenience macros
+==
+
+Two macros are provided that automatically release the read lock at the
+end of the scope.
+
+  RCU_READ_LOCK_GUARD()
+
+ Takes the lock and will release it at the end of the block it's
+ used in.
+
+  WITH_RCU_READ_LOCK_GUARD()  { code }
+
+ Is used at the head of a block to protect the code within the block.
+
+Note that 'goto'ing out of the guarded block will also drop the lock.
 
 DIFFERENCES WITH LINUX
 ==
diff --git a/include/qemu/rcu.h b/include/qemu/rcu.h
index 22876d1428..9c82683e37 100644
--- a/include/qemu/rcu.h
+++ b/include/qemu/rcu.h
@@ -154,6 +154,31 @@ extern void call_rcu1(struct rcu_head *head, RCUCBFunc 
*func);
   }),\
   (RCUCBFunc *)g_free);
 
+typedef void RCUReadAuto;
+static inline RCUReadAuto *rcu_read_auto_lock(void)
+{
+rcu_read_lock();
+/* Anything non-NULL causes the cleanup function to be called */
+return (void *)(uintptr_t)0x1;
+}
+
+static inline void rcu_read_auto_unlock(RCUReadAuto *r)
+{
+rcu_read_unlock();
+}
+
+G_DEFINE_AUTOPTR_CLEANUP_FUNC(RCUReadAuto, rcu_read_auto_unlock)
+
+#define WITH_RCU_READ_LOCK_GUARD() \
+WITH_RCU_READ_LOCK_GUARD_(_rcu_read_auto##__COUNTER__)
+
+#define WITH_RCU_READ_LOCK_GUARD_(var) \
+for (g_autoptr(RCUReadAuto) var = rcu_read_auto_lock(); \
+(var); rcu_read_auto_unlock(var), (var) = NULL)
+
+#define RCU_READ_LOCK_GUARD() \
+g_autoptr(RCUReadAuto) _rcu_read_auto __attribute__((unused)) = 
rcu_read_auto_lock()
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.23.0

[PULL 08/21] migration/postcopy: allocate tmp_page in setup stage

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

During migration, a tmp page is allocated so that we could place a whole
host page during postcopy.

Currently the page is allocated during load stage, this is a little bit
late. And more important, if we failed to allocate it, the error is not
checked properly. Even it is NULL, we would still use it.

This patch moves the allocation to setup stage and if failed error
message would be printed and caller would notice it.

Signed-off-by: Wei Yang 
Reviewed-by: Dr. David Alan Gilbert 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/postcopy-ram.c | 40 ++--
 migration/postcopy-ram.h |  7 ---
 migration/ram.c  |  2 +-
 3 files changed, 11 insertions(+), 38 deletions(-)

diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 1f63e65ed7..9a16f52a79 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -1134,6 +1134,16 @@ int postcopy_ram_enable_notify(MigrationIncomingState 
*mis)
 return -1;
 }
 
+mis->postcopy_tmp_page = mmap(NULL, mis->largest_page_size,
+  PROT_READ | PROT_WRITE, MAP_PRIVATE |
+  MAP_ANONYMOUS, -1, 0);
+if (mis->postcopy_tmp_page == MAP_FAILED) {
+mis->postcopy_tmp_page = NULL;
+error_report("%s: Failed to map postcopy_tmp_page %s",
+ __func__, strerror(errno));
+return -1;
+}
+
 /*
  * Ballooning can mark pages as absent while we're postcopying
  * that would cause false userfaults.
@@ -1260,30 +1270,6 @@ int postcopy_place_page_zero(MigrationIncomingState 
*mis, void *host,
 }
 }
 
-/*
- * Returns a target page of memory that can be mapped at a later point in time
- * using postcopy_place_page
- * The same address is used repeatedly, postcopy_place_page just takes the
- * backing page away.
- * Returns: Pointer to allocated page
- *
- */
-void *postcopy_get_tmp_page(MigrationIncomingState *mis)
-{
-if (!mis->postcopy_tmp_page) {
-mis->postcopy_tmp_page = mmap(NULL, mis->largest_page_size,
- PROT_READ | PROT_WRITE, MAP_PRIVATE |
- MAP_ANONYMOUS, -1, 0);
-if (mis->postcopy_tmp_page == MAP_FAILED) {
-mis->postcopy_tmp_page = NULL;
-error_report("%s: %s", __func__, strerror(errno));
-return NULL;
-}
-}
-
-return mis->postcopy_tmp_page;
-}
-
 #else
 /* No target OS support, stubs just fail */
 void fill_destination_postcopy_migration_info(MigrationInfo *info)
@@ -1341,12 +1327,6 @@ int postcopy_place_page_zero(MigrationIncomingState 
*mis, void *host,
 return -1;
 }
 
-void *postcopy_get_tmp_page(MigrationIncomingState *mis)
-{
-assert(0);
-return NULL;
-}
-
 int postcopy_wake_shared(struct PostCopyFD *pcfd,
  uint64_t client_addr,
  RAMBlock *rb)
diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h
index 9c8bd2bae0..fb4cd11d28 100644
--- a/migration/postcopy-ram.h
+++ b/migration/postcopy-ram.h
@@ -100,13 +100,6 @@ typedef enum {
 POSTCOPY_INCOMING_END
 } PostcopyState;
 
-/*
- * Allocate a page of memory that can be mapped at a later point in time
- * using postcopy_place_page
- * Returns: Pointer to allocated page
- */
-void *postcopy_get_tmp_page(MigrationIncomingState *mis);
-
 PostcopyState postcopy_state_get(void);
 /* Set the state and return the old state */
 PostcopyState postcopy_state_set(PostcopyState new_state);
diff --git a/migration/ram.c b/migration/ram.c
index e29c8b3408..071687ef37 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -4032,7 +4032,7 @@ static int ram_load_postcopy(QEMUFile *f)
 bool matches_target_page_size = false;
 MigrationIncomingState *mis = migration_incoming_get_current();
 /* Temporary page that is later 'placed' */
-void *postcopy_host_page = postcopy_get_tmp_page(mis);
+void *postcopy_host_page = mis->postcopy_tmp_page;
 void *last_host = NULL;
 bool all_zero = false;
 
-- 
2.23.0

[PULL 00/21] migration queue

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" 

The following changes since commit 98b2e3c9ab3abfe476a2b02f8f51813edb90e72d:

  Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into 
staging (2019-10-08 16:08:35 +0100)

are available in the Git repository at:

  git://github.com/dagrh/qemu.git tags/pull-migration-20191011a

for you to fetch changes up to 9a85e4b8f672016adbf7b7d5beaab2a99b9b5615:

  migration: Support gtree migration (2019-10-11 17:52:31 +0100)


Migration pull 2019-10-11

Mostly cleanups and minor fixes

[Note I'm seeing a hang on the aarch64 hosted x86-64 tcg migration
test in xbzrle; but I'm seeing that on current head as well]


Dr. David Alan Gilbert (6):
  rcu: Add automatically released rcu_read_lock variants
  migration: Fix missing rcu_read_unlock
  migration: Use automatic rcu_read unlock in ram.c
  migration: Use automatic rcu_read unlock in rdma.c
  rcu: Use automatic rc_read unlock in core memory/exec code
  migration: Don't try and recover return path in non-postcopy

Eric Auger (1):
  migration: Support gtree migration

Wei Yang (14):
  migration: use migration_is_active to represent active state
  migration/postcopy: allocate tmp_page in setup stage
  migration/postcopy: map large zero page in postcopy_ram_incoming_setup()
  migration/postcopy: fix typo in mark_postcopy_blocktime_begin's comment
  migration: pass in_postcopy instead of check state again
  migration: report SaveStateEntry id and name on failure
  migration/postcopy: mis->have_listen_thread check will never be touched
  migration/postcopy: postpone setting PostcopyState to END
  migration/postcopy: rename postcopy_ram_enable_notify to 
postcopy_ram_incoming_setup
  migration/postcopy: check PostcopyState before setting to 
POSTCOPY_INCOMING_RUNNING
  migration/multifd: fix a typo in comment of multifd_recv_unfill_packet()
  migration/multifd: use pages->allocated instead of the static max
  migration/multifd: initialize packet->magic/version once at setup stage
  migration/multifd: pages->used would be cleared when attach to 
multifd_send_state

 docs/devel/rcu.txt  |  16 ++
 exec.c  | 116 +---
 include/exec/ram_addr.h | 138 +++
 include/migration/misc.h|   1 +
 include/migration/vmstate.h |  40 +
 include/qemu/rcu.h  |  25 +++
 memory.c|  15 +-
 migration/migration.c   |  17 +-
 migration/postcopy-ram.c|  88 -
 migration/postcopy-ram.h|   9 +-
 migration/ram.c | 298 +++
 migration/rdma.c|  57 ++
 migration/savevm.c  |  14 +-
 migration/trace-events  |   5 +
 migration/vmstate-types.c   | 152 
 tests/test-vmstate.c| 421 
 16 files changed, 980 insertions(+), 432 deletions(-)

[PULL 07/21] migration: Don't try and recover return path in non-postcopy

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" 

In normal precopy we can't do reconnection recovery - but we also
don't need to, since you can just rerun migration.
At the moment if the 'return-path' capability is on, we use
the return path in precopy to give a positive 'OK' to the end
of migration; however if migration fails then we fall into
the postcopy recovery path and hang.  This fixes it by only
running the return path in the postcopy case.

Reported-by: Greg Kurz 
Tested-by: Greg Kurz 
Reviewed-by: Peter Xu 
Signed-off-by: Dr. David Alan Gilbert 
---
 migration/migration.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/migration/migration.c b/migration/migration.c
index 0c51aa6ac7..d7f8b428e0 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -2486,7 +2486,7 @@ retry:
 out:
 res = qemu_file_get_error(rp);
 if (res) {
-if (res == -EIO) {
+if (res == -EIO && migration_in_postcopy()) {
 /*
  * Maybe there is something we can do: it looks like a
  * network down issue, and we pause for a recovery.
-- 
2.23.0

[PULL 01/21] migration: use migration_is_active to represent active state

[Dr. David Alan Gilbert (git)]

From: Wei Yang 

Wrap the check into a function to make it easy to read.

Signed-off-by: Wei Yang 
Message-Id: <20190717005341.14140-1-richardw.y...@linux.intel.com>
Reviewed-by: Dr. David Alan Gilbert 
Signed-off-by: Dr. David Alan Gilbert 
---
 include/migration/misc.h |  1 +
 migration/migration.c| 12 
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/include/migration/misc.h b/include/migration/misc.h
index b9d8e787af..d2762257aa 100644
--- a/include/migration/misc.h
+++ b/include/migration/misc.h
@@ -60,6 +60,7 @@ void migration_object_init(void);
 void migration_shutdown(void);
 void qemu_start_incoming_migration(const char *uri, Error **errp);
 bool migration_is_idle(void);
+bool migration_is_active(MigrationState *);
 void add_migration_state_change_notifier(Notifier *notify);
 void remove_migration_state_change_notifier(Notifier *notify);
 bool migration_in_setup(MigrationState *);
diff --git a/migration/migration.c b/migration/migration.c
index 5f7e4d15e9..0c51aa6ac7 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1533,8 +1533,7 @@ static void migrate_fd_cleanup(MigrationState *s)
 qemu_fclose(tmp);
 }
 
-assert((s->state != MIGRATION_STATUS_ACTIVE) &&
-   (s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE));
+assert(!migration_is_active(s));
 
 if (s->state == MIGRATION_STATUS_CANCELLING) {
 migrate_set_state(>state, MIGRATION_STATUS_CANCELLING,
@@ -1703,6 +1702,12 @@ bool migration_is_idle(void)
 return false;
 }
 
+bool migration_is_active(MigrationState *s)
+{
+return (s->state == MIGRATION_STATUS_ACTIVE ||
+s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE);
+}
+
 void migrate_init(MigrationState *s)
 {
 /*
@@ -3266,8 +3271,7 @@ static void *migration_thread(void *opaque)
 
 trace_migration_thread_setup_complete();
 
-while (s->state == MIGRATION_STATUS_ACTIVE ||
-   s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) {
+while (migration_is_active(s)) {
 int64_t current_time;
 
 if (urgent || !qemu_file_rate_limit(s->to_dst_file)) {
-- 
2.23.0

Re: [RFC v5 067/126] block: introduce ERRP_AUTO_PROPAGATE

[Eric Blake]

On 10/11/19 11:04 AM, Vladimir Sementsov-Ogievskiy wrote:

If we want to add some info to errp (by error_prepend() or
error_append_hint()), we must use the ERRP_AUTO_PROPAGATE macro.
Otherwise, this info will not be added when errp == _err
(the program will exit prior to the error_append_hint() or
error_prepend() call).  Fix such cases.

If we want to check error after errp-function call, we need to
introduce local_err and than propagate it to errp. Instead, use
ERRP_AUTO_PROPAGATE macro, benefits are:
1. No need of explicit error_propagate call
2. No need of explicit local_err variable: use errp directly
3. ERRP_AUTO_PROPAGATE leaves errp as is if it's not NULL or
_fatel, this means that we don't break error_abort
(we'll abort on error_set, not on error_propagate)





Reported-by: Kevin Wolf 
Reported-by: Greg Kurz 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
---



  17 files changed, 287 insertions(+), 381 deletions(-)


A bit large, but still manageable for manually checking.


+++ b/block.c



@@ -1565,6 +1558,7 @@ fail_opts:
  
  static QDict *parse_json_filename(const char *filename, Error **errp)

  {
+ERRP_AUTO_PROPAGATE();
  QObject *options_obj;
  QDict *options;
  int ret;


Not shown in the diff, but this one is correct because of the use of 
error_prepend().


It does make me wonder if separating the error_prepend/error_hint vs. 
error_propagate into two separate cleanups (by incrementally amending 
the .cocci script between two series of generated patches) makes the 
overall review easier or harder.  But don't respin the series just 
because of me - wait for feedback from Markus as well on the best path 
forward.




+++ b/block/backup.c
@@ -583,6 +583,7 @@ static const BlockJobDriver backup_job_driver = {
  static int64_t backup_calculate_cluster_size(BlockDriverState *target,
   Error **errp)
  {
+ERRP_AUTO_PROPAGATE();
  int ret;
  BlockDriverInfo bdi;
  


Here, the diff is correct because of error_append_hint.


+++ b/block/dirty-bitmap.c
@@ -237,6 +237,7 @@ static bool bdrv_dirty_bitmap_recording(BdrvDirtyBitmap 
*bitmap)
  int bdrv_dirty_bitmap_check(const BdrvDirtyBitmap *bitmap, uint32_t flags,
  Error **errp)
  {
+ERRP_AUTO_PROPAGATE();
  if ((flags & BDRV_BITMAP_BUSY) && bdrv_dirty_bitmap_busy(bitmap)) {
  error_setg(errp, "Bitmap '%s' is currently in use by another"
 " operation and cannot be used", bitmap->name);


and here


+++ b/block/qapi.c
@@ -44,6 +44,7 @@
  BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
  BlockDriverState *bs, Error **errp)
  {
+ERRP_AUTO_PROPAGATE();
  ImageInfo **p_image_info;
  BlockDriverState *bs0;
  BlockDeviceInfo *info;
@@ -148,10 +149,8 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
  p_image_info = >image;
  info->backing_file_depth = 0;
  while (1) {
-Error *local_err = NULL;
-bdrv_query_image_info(bs0, p_image_info, _err);
-if (local_err) {
-error_propagate(errp, local_err);
+bdrv_query_image_info(bs0, p_image_info, errp);
+if (*errp) {
  qapi_free_BlockDeviceInfo(info);
  return NULL;
  }


I'm a bit wary of auto-propagate at the top level, but a local variable 
declared inside a loop - if the loop repeats more than once, we need to 
make sure that no subsequent iteration of the loop gets handed an 
already-set errp from an earlier iteration.  In this case, we're safe 
(the loop exits early due to return), but it's something to look out for 
in other patches.



+++ b/hw/block/onenand.c


A bit odd to group this with the others.  But not the end of the world.

Reviewed-by: Eric Blake 

So far, however, I have not tried rerunning the Coccinelle script, or 
checking (perhaps by sed) whether there are missed opportunities in 
these files that Coccinelle missed.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: Occasional VM soft lockup when a remote cdrom is attached

[John Snow]

On 10/11/19 9:22 AM, Guoheyi wrote:
> Hi folks,
> 
> We observed Linux on VM occasionally (at very low rate) got soft lockup
> when a remote cdrom is attached. The guest hangs up at below call trace:
> 

That's certainly a new one to me :)

> [Tue Oct8 23:02:53 2019]ata_scsi_queuecmd+0xe0/0x2a0 [libata]
> 
> [Tue Oct8 23:02:53 2019]scsi_dispatch_cmd+0xec/0x288
> 
> [Tue Oct8 23:02:53 2019]scsi_queue_rq+0x5f4/0x6b8
> 
> [Tue Oct8 23:02:53 2019]blk_mq_dispatch_rq_list+0xb0/0x690
> 
> [Tue Oct8 23:02:53 2019]blk_mq_do_dispatch_sched+0x8c/0x130
> 
> [Tue Oct8 23:02:53 2019]blk_mq_sched_dispatch_requests+0x128/0x1f0
> 
> [Tue Oct8 23:02:53 2019]__blk_mq_run_hw_queue+0x9c/0x128
> 
> [Tue Oct8 23:02:53 2019]__blk_mq_delay_run_hw_queue+0x198/0x1d8
> 
> [Tue Oct8 23:02:53 2019]blk_mq_run_hw_queue+0x68/0x180
> 
> [Tue Oct8 23:02:53 2019]blk_mq_sched_insert_request+0xbc/0x210
> 
> [Tue Oct8 23:02:53 2019]blk_execute_rq_nowait+0x118/0x168
> 
> [Tue Oct8 23:02:53 2019]blk_execute_rq+0x74/0xd8
> 
> [Tue Oct8 23:02:53 2019]__scsi_execute+0xd8/0x1e0
> 
> [Tue Oct8 23:02:53 2019]sr_check_events+0xd4/0x2c8 [sr_mod]
> 
> [Tue Oct8 23:02:53 2019]cdrom_check_events+0x34/0x50 [cdrom]
> 
> [Tue Oct8 23:02:53 2019]sr_block_check_events+0xdc/0x108 [sr_mod]
> 
> [Tue Oct8 23:02:53 2019]disk_check_events+0x60/0x198
> 
> [Tue Oct8 23:02:53 2019]disk_events_workfn+0x24/0x30
> 
> [Tue Oct8 23:02:53 2019]process_one_work+0x1b4/0x3f8
> 
> [Tue Oct8 23:02:53 2019]worker_thread+0x54/0x470
> 
> [Tue Oct8 23:02:53 2019]kthread+0x134/0x138
> 
> [Tue Oct8 23:02:53 2019]ret_from_fork+0x10/0x18
> 
> 
> We are running the whole stack on ARM64 platforms, using rcdrom on host
> to connect a remote cdrom, which is appeared as "/dev/sr0" on the host.
> Our Linux kernel version is 4.19.36 and qemu version is 2.8.1, which is
> fairly old but I checked the mainline and found the work flow does not
> change much. And KVM is enabled.
> 
> We provide the remote cdrom to guest as a block device, attached under
> ICH SATA bus.
> 
> 
> The work flow should be like this (please correct me if I was wrong):
> 
> 1. There is a kworker thread in guest kernel which will check cdrom
> status periodically.
> 
> 2. The call of "ata_scsi_queuecmd" in guest will write AHCI port
> register "PORT_CMD_ISSUE", so this VCPU thread is trapped out to qemu.
> 
> 3. qemu will grab the BQL and then dispatch the access to
> ahci_port_write().
> 
> 4. For this is a "get event status notification" command, qemu finally
> goes to cmd_get_event_status_notification() and then cdrom_is_inserted().
> 

via

cmd_get_event_status_notification (SCSI 0x4A)
  event_status_media
blk_is_inserted

> 5. In cdrom_is_inserted(), an ioctl to cdrom fd is issued.
> 

Using the bdrv_host_cdrom BlockDriver, for the .bdrv_is_inserted callback.

> 
> However, in the last step, we found the ioctl() may have large latency,
> for it is a virtual device of remote cdrom, when the remote server is
> busy and of poor performance. We have observed more than 8 seconds
> latency in half an hour test, and the latency might reach more than 20
> seconds when guest soft lockup occurred.
> 

I'm not sure what can be done here. the host_cdrom driver has a few
methods to query state (cdrom_is_inserted, cdrom_eject,
cdrom_lock_medium) and in general code is going to rely on
bdrv_is_inserted returning a truthful answer.

(I'm not sure we have callbacks established to tell when the backing
media we are ourselves relying on has gone away. Maybe it could be
added, but it's not there now. We could maybe cache the answer if we had
something reliable.)

You could always try using the host_device driver instead of the
host_cdrom one, which will just treat it as a "normal" block device
instead of a CDROM one, and doesn't use any cdrom specific ioctls. It
might avoid the costly call.

> 
> My question is, is there any way to get around of this issue? Does it
> make sense for qemu to setup an IO thread to issue this ioctl() and let
> the VCPU thread return to guest as soon as possible? Or it is kernel's
> responsibility to break up the long time ioctl() and return to user space?
> 

Yeah, I think you could probably try to make this change -- the code is
unfortunately very callback-hell-ish with poor abstraction boundaries,
but obviously the data transfer commands already defer to bottom halves.

(Ideally, I think every ATAPI command would just immediately defer to a
bottom half, but I don't know what effect that would have on callers of
ide_atapi_cmd. I'd have to audit it, and it's quite squiggly.)

"Patches welcome" as they say, but it's quite messy down there.

--js

> 
> Any comments or advice will be appreciated :)
> 
> 
> HG
> 
> 
> 
> 

Re: [Qemu-devel] [RFC 2 PATCH 13/16] machine: Add new epyc property in PCMachineState

[Eduardo Habkost]

On Fri, Oct 11, 2019 at 04:59:37PM +, Moger, Babu wrote:
> 
> On 10/10/19 10:59 PM, Eduardo Habkost wrote:
> > On Fri, Sep 06, 2019 at 07:13:09PM +, Moger, Babu wrote:
> >> Adds new epyc property in PCMachineState and also in MachineState.
> >> This property will be used to initialize the mode specific handlers
> >> to generate apic ids.
> >>
> >> Signed-off-by: Babu Moger 
> >> ---
> > [...]
> >> diff --git a/include/hw/boards.h b/include/hw/boards.h
> >> index 12eb5032a5..0001d42e50 100644
> >> --- a/include/hw/boards.h
> >> +++ b/include/hw/boards.h
> >> @@ -299,6 +299,8 @@ struct MachineState {
> >>  AccelState *accelerator;
> >>  CPUArchIdList *possible_cpus;
> >>  CpuTopology smp;
> >> +bool epyc;
> >> +
> > 
> > This won't scale at all when we start adding new CPU models with
> > different topology constraints.
> 
> Yes, I knew. This could cause scaling issues. Let me see if we could
> do anything different.
> 
> > 
> > I still have hope we can avoid having separate set of topology ID
> > functions (see my reply to "hw/386: Add new epyc mode topology
> 
> Yes. That was my hope too. Let me think thru this bit more. I will come
> back on this.

If you don't manage to use a common function in the next version,
it's not a big deal.  My main request is to make the calculations
easier to follow (e.g. avoiding any expression with more than two
terms, and always using an explicit "_per_*" suffix in all
variables and constants).

There's one possible problem I didn't realize yesterday: we might
need a mechanism to force a field width to be larger than
apicid_bitwidth_for_count(number_of_ids) (e.g. having 2 bits for
core ID even if there's only 1 or 2 cores per CCX).  Maybe the
solution is to add optional field width parameters to
X86CPUTopoInfo.

Then we could redefine the width functions like this:

static inline unsigned apicid_core_width(X86CPUTopoInfo *topo)
{
return MAX(apicid_bitwidth_for_count(topo->nr_cores), topo->min_core_bits);
}


Maybe we could replace the collection of fields with arrays to make all
calculations generic.  Untested example:

enum TopoField {
TOPO_FIELD_THREADS = 0,
TOPO_FIELD_CORES,
TOPO_FIELD_CCXS,  /* AMD */
TOPO_FIELD_DIES = TOPO_FIELD_CCX, /* Intel */
TOPO_FIELD_NODES,
TOPO_FIELD_PKG,
MAX_TOPO_FIELD,
};

struct TopoFieldDefinition {
/* Number of IDs at this level */
unsigned count;
/* Minimum number of APIC ID bits for this level */
unsigned min_width;
};

struct X86CPUTopoInfo
{
TopoFieldDefinition fields[MAX_TOPO_FIELD];
};

struct X85CPUTopoIDs
{
unsigned ids[MAX_TOPO_FIELD];
};

static inline unsigned apicid_field_width(const X86CPUTopoInfo *topo, TopoField 
field)
{
TopoFieldDefinition *def = >fields[field];
return MAX(apicid_bitwidth_for_count(def->count), def->min_width);
}

static inline unsigned apicid_field_offset(const X86CPUTopoInfo *topo, 
TopoField field)
{
if (field == 0) {
return 0;
}
return apicid_field_offset(topo, field - 1) + apic_id_field_width(topo, 
field - 1);
}


static inline apic_id_t apicid_from_topo_ids(const X86CPUTopoInfo *topo,
 const X86CPUTopoIDs *ids)
{
TopoField field;
apic_id_t r = 0;
for (field = 0; l < MAX_TOPO_FIELD; l++) {
unsigned offset = apicid_field_offset(topo, field);
unsigned width = apicid_field_width(topo, field);
assert(apicid_bitwidth_for_count(ids->ids[field] + 1) < 
apicid_field_width(topo, field));
r = deposit64(r, offset, width,  ids->ids[field];
}
return r;
}

static inline void x86_topo_ids_from_apicid(apic_id_t apicid,
const X86CPUTopoInfo *topo,
X86CPUTopoIDs *ids)
{
TopoField field;
for (field = 0; l < MAX_TOPO_FIELD; l++) {
unsigned offset = apicid_field_offset(topo, field);
unsigned width = apicid_field_width(topo, field);
ids->ids[field] = extract64(apicid, offset, width);
}
}

> 
> 
> > decoding functions").  But if we really have to create separate
> > functions, we can make them part of the CPU model table, not a
> > boolean machine property.
> > 

-- 
Eduardo

Re: [RFC v5 004/126] hmp: drop Error pointer indirection in hmp_handle_error

[Dr. David Alan Gilbert]

* Eric Blake (ebl...@redhat.com) wrote:
> On 10/11/19 11:03 AM, Vladimir Sementsov-Ogievskiy wrote:
> > We don't need Error **, as all callers pass local Error object, which
> > isn't used after the call. Use Error * instead.
> > 
> > Signed-off-by: Vladimir Sementsov-Ogievskiy 
> > ---
> >   include/monitor/hmp.h  |   2 +-
> >   dump/dump-hmp-cmds.c   |   4 +-
> >   hw/core/machine-hmp-cmds.c |   6 +-
> >   monitor/hmp-cmds.c | 155 ++---
> >   qdev-monitor.c |   4 +-
> >   qom/qom-hmp-cmds.c |   4 +-
> >   6 files changed, 87 insertions(+), 88 deletions(-)
> > 
> 
> > +++ b/dump/dump-hmp-cmds.c
> > @@ -32,7 +32,7 @@ void hmp_dump_guest_memory(Monitor *mon, const QDict 
> > *qdict)
> >   if (zlib + lzo + snappy + win_dmp > 1) {
> >   error_setg(, "only one of '-z|-l|-s|-w' can be set");
> > -hmp_handle_error(mon, );
> > +hmp_handle_error(mon, err);
> >   return;
> >   }
> 
> Probably not for this series, but would a patch to various HMP files to have
> a g_auto() sort of reporting on an error the moment it goes out of scope
> (rather than having to manually call hmp_handle_error() everywhere) make
> sense?

Maybe; although lets be a bit careful just how much we hide behind
g_auto.

Dave

> -- 
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.   +1-919-301-3226
> Virtualization:  qemu.org | libvirt.org
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK

RE: [PATCH v6 4/4] colo: Update Documentation for continuous replication

[Zhang, Chen]

> -Original Message-
> From: Lukas Straub 
> Sent: Saturday, October 12, 2019 12:01 AM
> To: Zhang, Chen 
> Cc: qemu-devel ; Jason Wang
> ; Wen Congyang ;
> Xie Changlong ; Kevin Wolf
> ; Max Reitz ; qemu-block
> 
> Subject: Re: [PATCH v6 4/4] colo: Update Documentation for continuous
> replication
> 
> On Thu, 10 Oct 2019 10:34:15 +
> "Zhang, Chen"  wrote:
> 
> > > -Original Message-
> > > From: Lukas Straub 
> > > Sent: Wednesday, October 9, 2019 11:17 PM
> > > To: Zhang, Chen 
> > > Cc: qemu-devel ; Jason Wang
> > > ; Wen Congyang
> ; Xie
> > > Changlong ; Kevin Wolf
> ;
> > > Max Reitz ; qemu-block  bl...@nongnu.org>
> > > Subject: Re: [PATCH v6 4/4] colo: Update Documentation for
> > > continuous replication
> > >
> > > On Wed, 9 Oct 2019 08:36:52 +
> > > "Zhang, Chen"  wrote:
> > >
> > > > > -Original Message-
> > > > > From: Lukas Straub 
> > > > > Sent: Saturday, October 5, 2019 9:06 PM
> > > > > To: qemu-devel 
> > > > > Cc: Zhang, Chen ; Jason Wang
> > > > > ; Wen Congyang
> > > ; Xie
> > > > > Changlong ; Kevin Wolf
> > > ;
> > > > > Max Reitz ; qemu-block  > > bl...@nongnu.org>
> > > > > Subject: [PATCH v6 4/4] colo: Update Documentation for
> > > > > continuous replication
> > > > >
> > > > > Document the qemu command-line and qmp commands for
> continuous
> > > > > replication
> > > > >
> > > > > Signed-off-by: Lukas Straub 
> > > > > ---
> > > > >  docs/COLO-FT.txt   | 213 +++--
> 
> > > > >  docs/block-replication.txt |  28 +++--
> > > > >  2 files changed, 174 insertions(+), 67 deletions(-)
> > > > >
> > > > > diff --git a/docs/COLO-FT.txt b/docs/COLO-FT.txt index
> > > > > ad24680d13..bc1a0ccb99 100644
> > > > > --- a/docs/COLO-FT.txt
> > > > > +++ b/docs/COLO-FT.txt
> > > > > @@ -145,35 +145,65 @@ The diagram just shows the main qmp
> > > command,
> > > > > you can get the detail  in test procedure.
> > > > >
> > > > > ...
> > > > >
> > > > > +Note: Here we are running both instances on the same Host for
> > > > > +testing, change the IP Addresses if you want to run it on two
> > > > > +Hosts. Initally
> > > > > +127.0.0.1 is the Primary Host and 127.0.0.2 is the Secondary Host.
> > > > > +
> > > > > +== Startup qemu ==
> > > > > +1. Primary:
> > > > > +Note: Initally, $imagefolder/primary.qcow2 needs to be copied
> > > > > +to all
> > > Hosts.
> > > > > +# imagefolder="/mnt/vms/colo-test-primary"
> > > > > +
> > > > > +# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m
> 512 -
> > > smp
> > > > > 1 -qmp stdio \
> > > > > +   -device piix3-usb-uhci -device usb-tablet -name primary \
> > > > > +   -netdev
> > > > > + tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper
> > > > > \
> > > > > +   -device rtl8139,id=e0,netdev=hn0 \
> > > > > +   -chardev socket,id=mirror0,host=0.0.0.0,port=9003,server,nowait \
> > > > > +   -chardev
> > > > > + socket,id=compare1,host=0.0.0.0,port=9004,server,wait \
> > > >
> > > > We should change the host=127.0.0.1 consistent with the expression
> below.
> > >
> > > Hi,
> > > This (and the IPs below in the QMP commands) needs to be this way,
> > > because it's a listening port and with 127.0.0.1 it would only
> > > listen on the loopback ip and wouldn't be reachable from another
> > > node for example. With
> > > 0.0.0.0 it will listen on all Interfaces.
> >
> > Yes, I know.  For this command demo, maybe use 192.168.0.1/192.168.0.2
> are more clear.
> 
> Hmm,
> the compare0 and compare_out actually can be replaced by unix sockets.
> So what do you think about the following?
> 
>-chardev socket,id=mirror0,host=127.0.0.1,port=9003,server,nowait \
>-chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
>-chardev socket,id=compare0,path=/tmp/compare0.sock,server,nowait \
>-chardev socket,id=compare0-0,path=/tmp/compare0.sock \
>-chardev
> socket,id=compare_out,path=/tmp/compare_out.sock,server,nowait \
>-chardev socket,id=compare_out0,path=/tmp/compare_out.sock \

In this way, user must create the unix socket node before start COLO, it is 
very hard to use.
I re-considered the issue here, looks keep the 0.0.0.0 is a relatively 
reasonably choice.
We can add some note here like you said before, to make sure user know the 
means of 0.0.0.0. 

Thanks
Zhang Chen

> 
> > >
> > > > > +   -chardev
> > > > > + socket,id=compare0,host=127.0.0.1,port=9001,server,nowait
> > > \
> > > > > +   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
> > > > > +   -chardev
> > > > > + socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait
> > > > > \
> > > > > +   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
> > > > > +   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0
> \
> > > > > +   -object filter-
> > > > > redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
> > > > > +   -object filter-
> > > > > redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
> > > > > +   -object iothread,id=iothread1 \

Re: [RFC v5 000/126] error: auto propagated local_err

[Eric Blake]

On 10/11/19 11:03 AM, Vladimir Sementsov-Ogievskiy wrote:

Hi all!

At the request of Markus: full version of errp propagation. Let's look
at it. Cover as much as possible, except inserting macro invocation
where it's not necessary.

It's huge, and so it's an RFC.


Is there a repo containing these patches, to make it easier to play with 
them locally without having to 'git am' the entire 126 messages?




  util/qemu-sockets.c   |  31 +--
  vl.c  |  14 +-
  python/commit-per-subsystem.py| 204 ++
  scripts/coccinelle/auto-propagated-errp.cocci | 118 
  341 files changed, 3851 insertions(+), 4455 deletions(-)
  create mode 100755 python/commit-per-subsystem.py
  create mode 100644 scripts/coccinelle/auto-propagated-errp.cocci


So, whether or not we take commit-per-subsystem.py, the overall series 
appears to be a nice reduction in lines of code.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [RFC v5 007/126] nbd: well form nbd_iter_channel_error errp handler

[Eric Blake]

On 10/11/19 11:03 AM, Vladimir Sementsov-Ogievskiy wrote:

Make nbd_iter_channel_error errp handler well formed:
rename errp to errp_in, as it is IN-parameter here (which is unusual


Actually, rename local_err to errp_in.


for errp).

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
  block/nbd.c | 10 +-
  1 file changed, 5 insertions(+), 5 deletions(-)



With commit message fixed,

Reviewed-by: Eric Blake 


  static void nbd_iter_channel_error(NBDReplyChunkIter *iter,
-   int ret, Error **local_err)
+   int ret, Error **errp_in)


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

ACPI table modifications

[Gautam Bhat]

Hi,

I want to add some I2C based temperature sensors to the -M Q35 machine. I
want to update the ACPI tables to add this device information. How can I go
about  doing this?

Thanks,
Gautam.