[Qemu-devel] [Bug 1516446] Re: Migration always causes guest freeze in one direction.

[Launchpad Bug Tracker]

[Expired for QEMU because there has been no activity for 60 days.]

** Changed in: qemu
   Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1516446

Title:
  Migration always causes guest freeze in one direction.

Status in QEMU:
  Expired

Bug description:
  Hello,

  I have three debian jessie machines standard installations except for
  homebuild qemu-2.4.0 package using the source package from testing. I
  had the same problem with the standard debian jessie qemu 2.1 too.

  I have host A, B and C.

  Migrations work between all combinations of these except A -> B. B ->
  A works.

  I use libvirt but as per your written request I have run qemu directly
  and verified the same problem.

  Host A:
  qemu-system-x86_64 --enable-kvm -name ashole -cpu kvm64 -m 1024 -drive 
file=/mnt/synctest/ashole.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none
 -vnc 0.0.0.0:600 -k sv -vga std -device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1

  Host B:
  qemu-system-x86_64 --enable-kvm -name ashole -cpu kvm64 -m 1024 -drive 
file=/mnt/synctest/ashole.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none
 -vnc 0.0.0.0:600 -k sv -vga std -device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
 -incoming tcp:0:

  Then in qemu monitor I run migrate -d tcp:B: and the guest freeze.

  I have tried with these guest os:es, freebsd 9.1, debian wheezy and
  debian jessie (Standard 3.16 kernel and backported 4.2 kernel), same
  problem with all of them. when running the migration through libvirt
  virt-manager says the guest is using 100% cpu.

  I had a similar problem
  (https://bugzilla.kernel.org/show_bug.cgi?id=61971) 2 years ago which
  was solved in kernel 3.13 if I remember correctly.

  Best Regards
  Magnus

  CPU info:
  Host A
  processor : 0
  vendor_id : AuthenticAMD
  cpu family: 21
  model : 2
  model name: AMD FX(tm)-8320 Eight-Core Processor
  stepping  : 0
  microcode : 0x600081f
  cpu MHz   : 1400.000
  cache size: 2048 KB
  physical id   : 0
  siblings  : 8
  core id   : 0
  cpu cores : 4
  apicid: 16
  initial apicid: 0
  fpu   : yes
  fpu_exception : yes
  cpuid level   : 13
  wp: yes
  flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb 
rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni 
pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c 
lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch 
osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core 
perfctr_nb arat cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean 
flushbyasid decodeassists pausefilter pfthreshold vmmcall bmi1
  bugs  : fxsave_leak sysret_ss_attrs
  bogomips  : 7023.54
  TLB size  : 1536 4K pages
  clflush size  : 64
  cache_alignment   : 64
  address sizes : 48 bits physical, 48 bits virtual
  power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

  processor : 1
  vendor_id : AuthenticAMD
  cpu family: 21
  model : 2
  model name: AMD FX(tm)-8320 Eight-Core Processor
  stepping  : 0
  microcode : 0x600081f
  cpu MHz   : 1400.000
  cache size: 2048 KB
  physical id   : 0
  siblings  : 8
  core id   : 1
  cpu cores : 4
  apicid: 17
  initial apicid: 1
  fpu   : yes
  fpu_exception : yes
  cpuid level   : 13
  wp: yes
  flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb 
rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni 
pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c 
lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch 
osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core 
perfctr_nb arat cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean 
flushbyasid decodeassists pausefilter pfthreshold vmmcall bmi1
  bugs  : fxsave_leak sysret_ss_attrs
  bogomips  : 7023.54
  TLB size  : 1536 4K pages
  clflush size  : 64
  cache_alignment   : 64
  address sizes : 48 bits physical, 48 bits virtual
  power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

  processor : 2
  vendor_id : AuthenticAMD
  cpu family: 21
  model : 2
  model name: AMD FX(tm)-8320 Eight-Core Processor
  stepping  : 0
  microcode : 0x600081f
  cpu MHz   : 1700.000
  cache size: 2048 KB
  physical id   

Re: [Qemu-devel] [PATCH v3 11/13] nvdimm: allow setting the label-size to 0

[Haozhong Zhang]

On 06/15/18 16:04, David Hildenbrand wrote:
> It is inititally 0, so setting it to 0 should be allowed, too.

I'm fine with this change and believe nothing is broken in practice,
but what is expected by the user who sets a zero label size?

Look at nvdimm_dsm_device() which enables label DSMs only if the label
size is not smaller than 128 KB. If a user sets a zero label size
explicitly, does he/she expect those label DSMs are available in
guest?  (according to Intel spec, the minimal label size is 128
KBytes)

I think if it's allowed to set a zero label-size, it would be better
to document its difference from other non-zero values in docs/nvdimm.txt.

Thanks,
Haozhong

> 
> Signed-off-by: David Hildenbrand 
> ---
>  hw/mem/nvdimm.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/mem/nvdimm.c b/hw/mem/nvdimm.c
> index db7d8c3050..df7646488b 100644
> --- a/hw/mem/nvdimm.c
> +++ b/hw/mem/nvdimm.c
> @@ -52,9 +52,9 @@ static void nvdimm_set_label_size(Object *obj, Visitor *v, 
> const char *name,
>  if (local_err) {
>  goto out;
>  }
> -if (value < MIN_NAMESPACE_LABEL_SIZE) {
> +if (value && value < MIN_NAMESPACE_LABEL_SIZE) {
>  error_setg(_err, "Property '%s.%s' (0x%" PRIx64 ") is required"
> -   " at least 0x%lx", object_get_typename(obj),
> +   " either 0 or at least 0x%lx", object_get_typename(obj),
> name, value, MIN_NAMESPACE_LABEL_SIZE);
>  goto out;
>  }
> -- 
> 2.17.1
> 
> 



signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH 6/6] linux-user: Use *at functions to implement interp_prefix

[Evan Green]

Apologies for getting lost in the dust storm, but where does this
leave this patch? Sounds like there's a real failure caused by it?
-Evan
On Thu, Jun 7, 2018 at 9:43 AM Richard Henderson
 wrote:
>
> On 06/07/2018 01:01 AM, Laurent Vivier wrote:
> >> Your script doesn't work outside debian, lacking debootstrap.
> >> At the moment, I can't build on debian *at all*.  Some bit of the build
> >> infrastructure is off and QEMU_FULL_VERSION gets incorrectly defined.  I 
> >> have
> >> no idea how or why it is different than Fedora.
> >
> > I had this problem on Fedora building in a separate directory, "make
> > distclean" at the top source directory fixed the problem for me.
>
> Thanks.  I had never seen this failure mode before.
>
>
> r~

Re: [Qemu-devel] [PATCH v7 00/54] fix building of tests/tcg - last chance to review!

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20180615194705.28019-1-alex.ben...@linaro.org
Subject: [Qemu-devel] [PATCH v7 00/54] fix building of tests/tcg - last chance 
to review!

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
failed=1
echo
fi
n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]   
patchew/20180615194705.28019-1-alex.ben...@linaro.org -> 
patchew/20180615194705.28019-1-alex.ben...@linaro.org
Switched to a new branch 'test'
ea7a4dbd0b .travis.yml: add check-tcg test
652c13f1eb tests/docker/Makefile.include: only force SID to NOCACHE if old
b70bc02b5f docker: docker.py adding age check command
fbdbc02b19 tests/Makefile: call sub-makes with SKIP_DOCKER_BUILD=1
5fc8f8b693 docker: docker.py add check sub-command
77bed125f5 docker: docker.py don't conflate checksums for extra_files
148852d075 tests: add top-level make dependency for docker builds
af6ee2ab56 tests/tcg: override runners for broken tests
7d4195d3d0 tests/tcg: add run, diff, and skip helper macros
432f648e13 tests/Makefile.include: add [build|clean|check]-tcg targets
08e57f8079 Makefile.target: add (clean-/build-)guest-tests targets
fccd475614 tests/tcg/Makefile: update to be called from Makefile.target
bd2f60e170 tests/tcg: enable building for PowerPC
b3ac2e1ba8 docker: move debian-powerpc-cross to sid based build
5cd0a65cbe tests/tcg: enable building for RISCV64
572ec17e38 tests/tcg: enable building for mips64
10fcf0b447 tests/tcg: enable building for sparc64
4a77005f2c tests/tcg: enable building for sh4
13f7f0d965 tests/tcg: enable building for m68k
3d20ff21ce tests/tcg: enable building for HPPA
ea2fcf674f tests/tcg/alpha: add Alpha specific tests
dd92f8fdfa tests/tcg: enable building for Alpha
a2d10caaf4 tests/tcg: enable building for ppc64
0e6da3e2c3 tests/tcg: enable building for s390x
f7c8cdc696 tests/tcg/mips: include common mips hello-mips
97518355bd tests/docker/Makefile.include: fix mipsel-cross dependancy
e359ed8d61 tests/tcg: enable building for MIPS
97ec8f3d2c tests/tcg: move MIPS specific tests into subdir
61086dc3e8 tests/tcg/arm: add fcvt test cases for AArch32/64
6cbb7a06c0 tests/tcg: enable building for AArch64
9a6f991ee2 tests/tcg/arm: fix up test-arm-iwmmxt test
32c5010c20 tests/tcg: enable building for ARM
2146fdbe57 tests/tcg: move ARM specific tests into subdir
3091d9d510 tests/tcg/i386/test-i386: fix printf format
b82c887f2a tests/tcg/i386/test-i386: use modern vector_size attributes
37aaf79acf tests/tcg/x86_64: add Makefile.target
b5c4e6bf40 tests/tcg/i386: add runner for test-i386-fprem
246d97eacf tests/tcg/i386: fix test-i386
8671ac9d73 tests/tcg/i386: Build fix for hello-i386
8edd5f2136 tests/tcg: enable building for i386
67af1a0edc tests/tcg: move i386 specific tests into subdir
de3b73900a tests/tcg/multiarch: move most output to stdout
3504d45b45 tests/tcg/multiarch: don't hard code paths/ports for linux-test
127f1a9428 tests/tcg/multiarch: enable additional linux-test tests
3e1df9f60f tests/tcg/multiarch: Build fix for linux-test
edc1f17df2 tests/tcg: move architecture independent tests into subdir
7e3c3cc8cd docker: Makefile.include introduce DOCKER_SCRIPT
4a9acb8a80 docker: allow "cc" command to run in user context
e578387528 docker: extend "cc" command to accept compiler
aecae6731d docker: Add "cc" subcommand
c9235deb15 configure: set cross_cc_FOO for host compiler
505c30b5c9 configure: allow user to specify --cross-cc-cflags-foo=
58d739bb59 configure: move i386_cc to cross_cc_i386
9944156f78 configure: add support for --cross-cc-FOO

=== OUTPUT BEGIN ===
Checking PATCH 1/54: configure: add support for --cross-cc-FOO...
Checking PATCH 2/54: configure: move i386_cc to cross_cc_i386...
Checking PATCH 3/54: configure: allow user to specify --cross-cc-cflags-foo=...
Checking PATCH 4/54: configure: set cross_cc_FOO for host compiler...
Checking PATCH 5/54: docker: Add "cc" subcommand...
Checking PATCH 6/54: docker: extend "cc" command to accept compiler...
Checking PATCH 7/54: docker: allow "cc" command to run in user context...
Checking PATCH 8/54: docker: Makefile.include introduce DOCKER_SCRIPT...
Checking PATCH 9/54: tests/tcg: move architecture independent tests into 
subdir...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#138: 
new file mode 100644

total: 0 errors, 1 warnings, 136 lines checked

Your patch has style problems, please review.  If 

[Qemu-devel] [PATCH v3 2/2] i386/cpu: make -cpu host support monitor/mwait

[Michael S. Tsirkin]

When guest CPU PM is enabled, and with -cpu host, expose the host CPU
MWAIT leaf in the CPUID so guest can make good PM decisions.

Note: the result is 100% CPU utilization reported by host as host
no longer knows that the CPU is halted.

Signed-off-by: Michael S. Tsirkin 
---
 target/i386/cpu.h |  9 +
 target/i386/cpu.c | 19 ++-
 target/i386/kvm.c |  9 +
 3 files changed, 32 insertions(+), 5 deletions(-)

diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 664504610e..309f804573 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -1378,6 +1378,15 @@ struct X86CPU {
 /* if true the CPUID code directly forward host cache leaves to the guest 
*/
 bool cache_info_passthrough;
 
+/* if true the CPUID code directly forwards
+ * host monitor/mwait leaves to the guest */
+struct {
+uint32_t eax;
+uint32_t ebx;
+uint32_t ecx;
+uint32_t edx;
+} mwait;
+
 /* Features that were filtered out because of missing host capabilities */
 uint32_t filtered_features[FEATURE_WORDS];
 
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 94260412e2..a4fb856d58 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -3760,11 +3760,11 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, 
uint32_t count,
 }
 break;
 case 5:
-/* mwait info: needed for Core compatibility */
-*eax = 0; /* Smallest monitor-line size in bytes */
-*ebx = 0; /* Largest monitor-line size in bytes */
-*ecx = CPUID_MWAIT_EMX | CPUID_MWAIT_IBE;
-*edx = 0;
+/* MONITOR/MWAIT Leaf */
+*eax = cpu->mwait.eax; /* Smallest monitor-line size in bytes */
+*ebx = cpu->mwait.ebx; /* Largest monitor-line size in bytes */
+*ecx = cpu->mwait.ecx; /* flags */
+*edx = cpu->mwait.edx; /* mwait substates */
 break;
 case 6:
 /* Thermal and Power Leaf */
@@ -4595,6 +4595,15 @@ static void x86_cpu_realizefn(DeviceState *dev, Error 
**errp)
 goto out;
 }
 
+if (xcc->host_cpuid_required && enable_cpu_pm) {
+host_cpuid(5, 0, >mwait.eax, >mwait.ebx,
+   >mwait.ecx, >mwait.edx);
+env->features[FEAT_1_ECX] |= CPUID_EXT_MONITOR;
+}
+/* mwait extended info: needed for Core compatibility */
+/* We always wake on interrupt even if host does not have the capability */
+cpu->mwait.ecx |= CPUID_MWAIT_EMX | CPUID_MWAIT_IBE;
+
 if (cpu->apic_id == UNASSIGNED_APIC_ID) {
 error_setg(errp, "apic-id property was not initialized properly");
 return;
diff --git a/target/i386/kvm.c b/target/i386/kvm.c
index cf9107be4b..805968d5b7 100644
--- a/target/i386/kvm.c
+++ b/target/i386/kvm.c
@@ -366,6 +366,15 @@ uint32_t kvm_arch_get_supported_cpuid(KVMState *s, 
uint32_t function,
 if (!kvm_irqchip_in_kernel()) {
 ret &= ~CPUID_EXT_X2APIC;
 }
+
+if (enable_cpu_pm) {
+int disable_exits = kvm_check_extension(s,
+KVM_CAP_X86_DISABLE_EXITS);
+
+if (disable_exits & KVM_X86_DISABLE_EXITS_MWAIT) {
+ret |= CPUID_EXT_MONITOR;
+}
+}
 } else if (function == 6 && reg == R_EAX) {
 ret |= CPUID_6_EAX_ARAT; /* safe to allow because of emulated APIC */
 } else if (function == 7 && index == 0 && reg == R_EBX) {
-- 
MST

[Qemu-devel] [PATCH v3 0/2] kvm: limited x86 CPU power management

[Michael S. Tsirkin]

This adds ability to expose some host CPU power management capabilities
to guests. For intel guests, this is sufficient for guest to enable low
power CPU states on idle. For AMD guests it isn't sufficient, deeper
C-states are entered using System-IO.

When enabled this puts CPU in a low power state with exit latencies that
can go up to multiple milliseconds, and makes host scheduler as well as
host utilities such as top and powertop think the CPU is constantly
busy.  Thus it has the effect of dedicating a host CPU for this guest.

mwait based power management is tied closely to specifics of CPUID,
making migration challenging. At this point only the non-migrateable
-cpu host is supported.

With this patch applied, VM latency is within the noise of
baremetal for some benchmarks.

perf bench sched pipe results:
Before:
6.452 sec
After:
4.382 sec
Baremetal:
4.136 sec

Changes since v2:
At Daniel's suggestion, don't use the -realtime flag.

At Paolo's suggestion, group this with memory lock flag
which has a similar effect of dedicating memory to this VM.

Michael S. Tsirkin (2):
  kvm: support -dedicated cpu-pm=on|off
  i386/cpu: make -cpu host support monitor/mwait

 include/sysemu/sysemu.h |  1 +
 target/i386/cpu.h   |  9 +
 target/i386/cpu.c   | 19 ++-
 target/i386/kvm.c   | 32 
 vl.c| 32 +++-
 qemu-options.hx | 18 ++
 6 files changed, 105 insertions(+), 6 deletions(-)

-- 
MST

[Qemu-devel] [PATCH v3 1/2] kvm: support -dedicated cpu-pm=on|off

[Michael S. Tsirkin]

With this flag, kvm allows guest to control host CPU power state.  This
increases latency for other processes using same host CPU in an
unpredictable way, but if decreases idle entry/exit times for the
running VCPU, so it works best if you use a dedicated host cpu,
hence the name.

Follow-up patches will expose this capability to guest
(using mwait leaf).

Based on a patch by Wanpeng Li  .

Signed-off-by: Michael S. Tsirkin 
---
 include/sysemu/sysemu.h |  1 +
 target/i386/kvm.c   | 23 +++
 vl.c| 32 +++-
 qemu-options.hx | 18 ++
 4 files changed, 73 insertions(+), 1 deletion(-)

diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index e893f72f3b..b921c6f3b7 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -128,6 +128,7 @@ extern bool boot_strict;
 extern uint8_t *boot_splash_filedata;
 extern size_t boot_splash_filedata_size;
 extern bool enable_mlock;
+extern bool enable_cpu_pm;
 extern uint8_t qemu_extra_params_fw[2];
 extern QEMUClockType rtc_clock;
 extern const char *mem_path;
diff --git a/target/i386/kvm.c b/target/i386/kvm.c
index 44f70733e7..cf9107be4b 100644
--- a/target/i386/kvm.c
+++ b/target/i386/kvm.c
@@ -1357,6 +1357,29 @@ int kvm_arch_init(MachineState *ms, KVMState *s)
 smram_machine_done.notify = register_smram_listener;
 qemu_add_machine_init_done_notifier(_machine_done);
 }
+
+if (enable_cpu_pm) {
+int disable_exits = kvm_check_extension(s, KVM_CAP_X86_DISABLE_EXITS);
+int ret;
+
+/* Work around for kernel header with a typo. TODO: fix header and drop. */
+#if defined(KVM_X86_DISABLE_EXITS_HTL) && !defined(KVM_X86_DISABLE_EXITS_HLT)
+#define KVM_X86_DISABLE_EXITS_HLT KVM_X86_DISABLE_EXITS_HTL
+#endif
+if (disable_exits) {
+disable_exits &= (KVM_X86_DISABLE_EXITS_MWAIT |
+  KVM_X86_DISABLE_EXITS_HLT |
+  KVM_X86_DISABLE_EXITS_PAUSE);
+}
+
+ret = kvm_vm_enable_cap(s, KVM_CAP_X86_DISABLE_EXITS, 0,
+disable_exits);
+if (ret < 0) {
+error_report("kvm: guest stopping CPU not supported: %s",
+ strerror(-ret));
+}
+}
+
 return 0;
 }
 
diff --git a/vl.c b/vl.c
index 06031715ac..d53a9abcde 100644
--- a/vl.c
+++ b/vl.c
@@ -142,6 +142,7 @@ ram_addr_t ram_size;
 const char *mem_path = NULL;
 int mem_prealloc = 0; /* force preallocation of physical target memory */
 bool enable_mlock = false;
+bool enable_cpu_pm = false;
 int nb_nics;
 NICInfo nd_table[MAX_NICS];
 int autostart;
@@ -390,6 +391,22 @@ static QemuOptsList qemu_realtime_opts = {
 },
 };
 
+static QemuOptsList qemu_dedicated_opts = {
+.name = "dedicated",
+.head = QTAILQ_HEAD_INITIALIZER(qemu_dedicated_opts.head),
+.desc = {
+{
+.name = "mem-lock",
+.type = QEMU_OPT_BOOL,
+},
+{
+.name = "cpu-pm",
+.type = QEMU_OPT_BOOL,
+},
+{ /* end of list */ }
+},
+};
+
 static QemuOptsList qemu_msg_opts = {
 .name = "msg",
 .head = QTAILQ_HEAD_INITIALIZER(qemu_msg_opts.head),
@@ -3903,7 +3920,20 @@ int main(int argc, char **argv, char **envp)
 if (!opts) {
 exit(1);
 }
-enable_mlock = qemu_opt_get_bool(opts, "mlock", true);
+/* Don't override the -dedicated option if set */
+enable_mlock = enable_mlock ||
+qemu_opt_get_bool(opts, "mlock", true);
+break;
+case QEMU_OPTION_dedicated:
+opts = qemu_opts_parse_noisily(qemu_find_opts("dedicated"),
+   optarg, false);
+if (!opts) {
+exit(1);
+}
+/* Don't override the -realtime option if set */
+enable_mlock = enable_mlock ||
+qemu_opt_get_bool(opts, "mem-lock", false);
+enable_cpu_pm = qemu_opt_get_bool(opts, "cpu-pm", false);
 break;
 case QEMU_OPTION_msg:
 opts = qemu_opts_parse_noisily(qemu_find_opts("msg"), optarg,
diff --git a/qemu-options.hx b/qemu-options.hx
index c0d3951e9f..ddedb7eb92 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -3337,6 +3337,24 @@ mlocking qemu and guest memory can be enabled via 
@option{mlock=on}
 (enabled by default).
 ETEXI
 
+DEF("dedicated", HAS_ARG, QEMU_OPTION_dedicated,
+"-dedicated [mem-lock=on|off][cpu-pm=on|off]\n"
+"run qemu with realtime features\n"
+"mem-lock=on|off controls memory lock support (default: 
off)\n"
+"cpu-pm=on|off controls cpu power management (default: 
off)\n",
+QEMU_ARCH_ALL)
+STEXI
+@item -dedicated mem-lock=on|off
+@item -dedicated cpu-pm=on|off

[Qemu-devel] [PATCH v3 1/2] qapi: open files in binary mode and use explicit decoding/encoding in common.py

[Matthias Maier]

This is a different approach to fix the locale dependent encode/decode
problem in common.py utilizing the binary read/write mode [1,2], and (if
a python 3 interpreter is used) with explicit decode/encode arguments
[3].

This approach is preferred over the fix in commit d4e5ec877ca because it
is (a) locale independent, and (b) does not depend on the en_US.UTF_8
locale to be available.

[1] https://docs.python.org/3.6/library/stdtypes.html#bytes.decode
[2] https://docs.python.org/3.6/library/stdtypes.html#str.encode
[3] https://docs.python.org/3/howto/unicode.html

Signed-off-by: Arfrever Frehtes Taifersar Arahesis 
Signed-off-by: Matthias Maier 
---
 scripts/qapi/common.py | 11 ---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 2462fc0291..a368e11a38 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -16,6 +16,7 @@ import errno
 import os
 import re
 import string
+import sys
 from collections import OrderedDict
 
 builtin_types = {
@@ -259,6 +260,8 @@ class QAPISchemaParser(object):
 previously_included.append(os.path.abspath(fp.name))
 self.incl_info = incl_info
 self.src = fp.read()
+if sys.version_info[0] >= 3:
+self.src = self.src.decode()
 if self.src == '' or self.src[-1] != '\n':
 self.src += '\n'
 self.cursor = 0
@@ -340,7 +343,7 @@ class QAPISchemaParser(object):
 return None
 
 try:
-fobj = open(incl_fname, 'r')
+fobj = open(incl_fname, 'rb')
 except IOError as e:
 raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname))
 return QAPISchemaParser(fobj, previously_included, info)
@@ -1492,7 +1495,7 @@ class QAPISchemaEvent(QAPISchemaEntity):
 class QAPISchema(object):
 def __init__(self, fname):
 self._fname = fname
-parser = QAPISchemaParser(open(fname, 'r'))
+parser = QAPISchemaParser(open(fname, 'rb'))
 exprs = check_exprs(parser.exprs)
 self.docs = parser.docs
 self._entity_list = []
@@ -2006,9 +2009,11 @@ class QAPIGen(object):
 if e.errno != errno.EEXIST:
 raise
 fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666)
-f = os.fdopen(fd, 'r+')
+f = os.fdopen(fd, 'r+b')
 text = (self._top(fname) + self._preamble + self._body
 + self._bottom(fname))
+if sys.version_info[0] >= 3:
+text = text.encode()
 oldtext = f.read(len(text) + 1)
 if text != oldtext:
 f.seek(0)
-- 
2.16.4

[Qemu-devel] [PATCH v3 2/2] Revert commit d4e5ec877ca

[Matthias Maier]

This commit removes the PYTHON_UTF8 workaround. The problem with setting

  LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8

is that the en_US.UTF-8 locale might not be available. In this case
setting above locales results in build errors even though another UTF-8
locale was originally set [1]. The only stable way of fixing the
encoding problem is by explicitly annotating encoding/decoding in the
python script.

[1] https://bugs.gentoo.org/657766

Signed-off-by: Arfrever Frehtes Taifersar Arahesis 
Signed-off-by: Matthias Maier 
---
 Makefile   | 6 ++
 tests/Makefile.include | 6 +++---
 2 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/Makefile b/Makefile
index e46f2b625a..7ed9cc4a21 100644
--- a/Makefile
+++ b/Makefile
@@ -20,8 +20,6 @@ ifneq ($(wildcard config-host.mak),)
 all:
 include config-host.mak
 
-PYTHON_UTF8 = LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 $(PYTHON)
-
 git-submodule-update:
 
 .PHONY: git-submodule-update
@@ -576,7 +574,7 @@ qga/qapi-generated/qga-qapi-commands.h 
qga/qapi-generated/qga-qapi-commands.c \
 qga/qapi-generated/qga-qapi-doc.texi: \
 qga/qapi-generated/qapi-gen-timestamp ;
 qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json 
$(qapi-py)
-   $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
+   $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
-o qga/qapi-generated -p "qga-" $<, \
"GEN","$(@:%-timestamp=%)")
@>$@
@@ -676,7 +674,7 @@ qapi/qapi-introspect.h qapi/qapi-introspect.c \
 qapi/qapi-doc.texi: \
 qapi-gen-timestamp ;
 qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
-   $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
+   $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
-o "qapi" -b $<, \
"GEN","$(@:%-timestamp=%)")
@>$@
diff --git a/tests/Makefile.include b/tests/Makefile.include
index ca91da26cb..88f1bc1242 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -678,13 +678,13 @@ tests/test-qapi-events.c tests/test-qapi-events.h \
 tests/test-qapi-introspect.c tests/test-qapi-introspect.h: \
 tests/test-qapi-gen-timestamp ;
 tests/test-qapi-gen-timestamp: 
$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(qapi-py)
-   $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
+   $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
-o tests -p "test-" $<, \
"GEN","$(@:%-timestamp=%)")
@>$@
 
 tests/qapi-schema/doc-good.test.texi: 
$(SRC_PATH)/tests/qapi-schema/doc-good.json $(qapi-py)
-   $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
+   $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
-o tests/qapi-schema -p "doc-good-" $<, \
"GEN","$@")
@mv tests/qapi-schema/doc-good-qapi-doc.texi $@
@@ -942,7 +942,7 @@ check-tests/qemu-iotests-quick.sh: 
tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: 
$(SRC_PATH)/%.json
$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-   $(PYTHON_UTF8) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+   $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
$^ >$*.test.out 2>$*.test.err; \
echo $$? >$*.test.exit, \
"TEST","$*.out")
-- 
2.16.4

[Qemu-devel] [PATCH v3 0/2] Fix compilation with python-3 if en_US.UTF-8 is unavailable

[Matthias Maier]

Hi,

this is hopefully the final iteration of the patches.

v3:
 - reverse order of patches
 - rename second patch to "Revert commit ..."
 - drop "UTF-8" argument in decode()/encode(); bot functions default to
   utf-8 already.

v2:
 - ensure compatibility with python2 by only calling encode()/decode() if
   script is run with a python3 interpreter

Best,
Matthias

Re: [Qemu-devel] [PATCH v2 2/2] qapi: open files in binary mode and use explicit decoding/encoding in common.py

[Matthias Maier]

On Fri, Jun 15, 2018, at 10:31 CDT, Markus Armbruster  wrote:

> If I understand 7.2.3. Standard Encodings[*] correctly, the canonical
> name is "utf-8".  Let's use that.  Wait, it's the default, no need to
> pass an argument.

Roger. I will change this in v3.

Re: [Qemu-devel] [PATCH v2 0/2] kvm: x86 CPU power management

[Michael S. Tsirkin]

On Fri, Jun 15, 2018 at 03:32:27PM -0300, Marcelo Tosatti wrote:
> On Thu, Jun 14, 2018 at 02:37:28AM +0300, Michael S. Tsirkin wrote:
> > On Wed, Jun 13, 2018 at 07:34:53PM -0300, Marcelo Tosatti wrote:
> > > On Tue, Jun 12, 2018 at 09:47:11PM +0300, Michael S. Tsirkin wrote:
> > > > This adds ability to expose host CPU power management capabilities to
> > > > guests. For intel guests, this is sufficient for guest to enable
> > > > low power CPU power management. For AMD guests it isn't sufficient,
> > > > deeper C-states are entered using System-IO.
> > > > 
> > > > mwait based power management is tied closely to specifics of CPUID,
> > > > making migration challenging. At this point only the non-migrateable
> > > > -cpu host is supported.
> > > > 
> > > > With this patch applied, VM latency is within the noise of
> > > > baremetal for some benchmarks.
> > > > 
> > > > perf bench sched pipe results:
> > > > Before:
> > > > 6.452 sec
> > > > After:
> > > > 4.382 sec
> > > > Baremetal:
> > > > 4.136 sec
> > > > 
> > > > Michael S. Tsirkin (2):
> > > >   kvm: support -realtime cpu-pm=on|off
> > > >   i386/cpu: make -cpu host support monitor/mwait
> > > > 
> > > >  include/sysemu/sysemu.h |  1 +
> > > >  target/i386/cpu.h   |  9 +
> > > >  target/i386/cpu.c   | 19 ++-
> > > >  target/i386/kvm.c   | 30 ++
> > > >  vl.c|  6 ++
> > > >  qemu-options.hx |  9 +++--
> > > >  6 files changed, 67 insertions(+), 7 deletions(-)
> > > > 
> > > > -- 
> > > > MST
> > > 
> > > Hi Michael,
> > > 
> > > 1) Command line option interface
> > > 
> > > Why is this not an optional cpu feature such as the other features? 
> > > 
> > > 
> > > -cpu CPU,+mwait  
> > > 
> > > rather than a separate, architecture independent "-realtime 
> > > cpu-pm=on|off" 
> > > command line option?
> > 
> > Because it's not just a guest flag. With guest pm on, one guest
> > can severely affect the latency of others on the same host CPU.
> 
> How so ?

Look at drivers/idle/intel_idle.c
There are states with exit latencies of 
1

> > > 2) Migration
> > > 
> > > Isnt it sufficient to check that both CPUID leafs are the same, 
> > > to allow migration ?
> > 
> > Not at the moment since linux guests use mwait hints and latency values
> > from a table in intel_idle.  If the host and guest models do not match,
> > mwait will get a wrong hint.
> > 
> > It will not do the right thing then!
> > 
> > You want exactly the same host CPU for it to work.
> > 
> > This isn't different from how -host cpu works generally.
> 
> Ok, makes sense.
> 
> > > 1. Check that the processor supports MONITOR and MWAIT. If
> > > CPUID.01H:ECX.MONITOR[bit 3] = 1, MONITOR and MWAIT are available at
> > > ring 0.
> > > 
> > > 2. Query the smallest and largest line size that MONITOR uses.
> > > Use CPUID.05H:EAX.smallest[bits 15:0];EBX.largest[bits15:0]. 
> > > 

Re: [Qemu-devel] [PATCH 1/2] Partially revert commit d4e5ec877ca

[Matthias Maier]

On Fri, Jun 15, 2018, at 15:34 CDT, Eduardo Habkost  wrote:

> Thanks for the patches.  It's nice to see that hack go away.
>
> Patch 2/2 is necessary to make the removal of PYTHON_UTF8 safe,
> correct?  In that case, I plan to reverse the order of the
> patches before applying.

Yes, the order of the patches should be reversed.

Re: [Qemu-devel] [PATCH 1/2] Partially revert commit d4e5ec877ca

[Eduardo Habkost]

On Thu, Jun 14, 2018 at 10:43:56AM -0500, Matthias Maier wrote:
> This commit removes the PYTHON_UTF8 workaround. The problem with setting
> 
>   LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8
> 
> is that the en_US.UTF-8 locale might not be available. In this case
> setting above locales results in build errors even though another UTF-8
> locale was originally set [1]. The only stable way of fixing the
> encoding problem is by explicitly annotating encoding/decoding in the
> python script.

Thanks for the patches.  It's nice to see that hack go away.

Patch 2/2 is necessary to make the removal of PYTHON_UTF8 safe,
correct?  In that case, I plan to reverse the order of the
patches before applying.

> 
> [1] https://bugs.gentoo.org/657766
[...]

-- 
Eduardo

[Qemu-devel] [PATCH v7 52/54] docker: docker.py adding age check command

[Alex Bennée]

This is useful for querying if an image is too old.

Signed-off-by: Alex Bennée 
---
 tests/docker/docker.py | 49 --
 1 file changed, 38 insertions(+), 11 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 3666197a16..ad0c47921a 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -29,6 +29,7 @@ from tarfile import TarFile, TarInfo
 from StringIO import StringIO
 from shutil import copy, rmtree
 from pwd import getpwuid
+from datetime import datetime,timedelta
 
 
 FILTERED_ENV_NAMES = ['ftp_proxy', 'http_proxy', 'https_proxy']
@@ -184,6 +185,10 @@ class Docker(object):
 labels = json.loads(resp)[0]["Config"].get("Labels", {})
 return labels.get("com.qemu.dockerfile-checksum", "")
 
+def get_image_creation_time(self, tag):
+resp = self._output(["inspect", tag])
+return json.loads(resp)[0]["Created"]
+
 def build_image(self, tag, docker_dir, dockerfile,
 quiet=True, user=False, argv=None, extra_files_cksum=[]):
 if argv == None:
@@ -450,21 +455,43 @@ class CheckCommand(SubCommand):
 def args(self, parser):
 parser.add_argument("tag",
 help="Image Tag")
-parser.add_argument("dockerfile",
-help="Dockerfile name")
+parser.add_argument("dockerfile", default=None,
+help="Dockerfile name", nargs='?')
+parser.add_argument("--checktype", choices=["checksum", "age"],
+default="checksum", help="check type")
+parser.add_argument("--olderthan", default=60, type=int,
+help="number of minutes")
 
 def run(self, args, argv):
-dockerfile = open(args.dockerfile, "rb").read()
 tag = args.tag
-
 dkr = Docker()
-if dkr.image_matches_dockerfile(tag, dockerfile):
-if not args.quiet:
-print("Image is up to date.")
-return 0
-else:
-print("Image needs updating")
-return 1
+
+if args.checktype == "checksum":
+if not args.dockerfile:
+print("Need a dockerfile for tag:%s" % (tag))
+return 1
+
+dockerfile = open(args.dockerfile, "rb").read()
+
+if dkr.image_matches_dockerfile(tag, dockerfile):
+if not args.quiet:
+print("Image is up to date")
+return 0
+else:
+print("Image needs updating")
+return 1
+elif args.checktype == "age":
+timestr = dkr.get_image_creation_time(tag).split(".")[0]
+created = datetime.strptime(timestr, "%Y-%m-%dT%H:%M:%S")
+past = datetime.now() - timedelta(minutes=args.olderthan)
+if created < past:
+print ("Image created @ %s more than %d minutes old" %
+   (timestr, args.olderthan))
+return 1
+else:
+if not args.quiet:
+print ("Image less than %d minutes old" % (args.olderthan))
+return 0
 
 
 def main():
-- 
2.17.1

[Qemu-devel] [PATCH v7 38/54] tests/tcg: enable building for sparc64

[Alex Bennée]

As before, using Debian SID compilers.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v5
   - add Makefile with EXTRA_RUNS for mmap tests
---
 tests/docker/Makefile.include|  1 +
 tests/docker/dockerfiles/debian-sparc64-cross.docker | 12 
 tests/tcg/sparc64/Makefile.include   |  2 ++
 tests/tcg/sparc64/Makefile.target|  6 ++
 4 files changed, 21 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-sparc64-cross.docker
 create mode 100644 tests/tcg/sparc64/Makefile.include
 create mode 100644 tests/tcg/sparc64/Makefile.target

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index a2dd307a8e..fb465e729e 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -74,6 +74,7 @@ docker-image-debian-alpha-cross: docker-image-debian-sid
 docker-image-debian-hppa-cross: docker-image-debian-sid
 docker-image-debian-m68k-cross: docker-image-debian-sid
 docker-image-debian-sh4-cross: docker-image-debian-sid
+docker-image-debian-sparc64-cross: docker-image-debian-sid
 docker-image-travis: NOUSER=1
 
 # Specialist build images, sometimes very limited tools
diff --git a/tests/docker/dockerfiles/debian-sparc64-cross.docker 
b/tests/docker/dockerfiles/debian-sparc64-cross.docker
new file mode 100644
index 00..1e2c809274
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-sparc64-cross.docker
@@ -0,0 +1,12 @@
+#
+# Docker cross-compiler target
+#
+# This docker target builds on the debian sid base image which
+# contains cross compilers for Debian "ports" targets.
+#
+FROM qemu:debian-sid
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+apt-get install -y --no-install-recommends \
+gcc-sparc64-linux-gnu \
+libc6-dev-sparc64-cross
diff --git a/tests/tcg/sparc64/Makefile.include 
b/tests/tcg/sparc64/Makefile.include
new file mode 100644
index 00..95fc8dee9f
--- /dev/null
+++ b/tests/tcg/sparc64/Makefile.include
@@ -0,0 +1,2 @@
+DOCKER_IMAGE=debian-sparc64-cross
+DOCKER_CROSS_COMPILER=sparc64-linux-gnu-gcc
diff --git a/tests/tcg/sparc64/Makefile.target 
b/tests/tcg/sparc64/Makefile.target
new file mode 100644
index 00..408dace783
--- /dev/null
+++ b/tests/tcg/sparc64/Makefile.target
@@ -0,0 +1,6 @@
+# -*- Mode: makefile -*-
+#
+# sparc specific tweaks
+
+# On Sparc64 Linux support 8k pages
+EXTRA_RUNS+=run-test-mmap-8192
-- 
2.17.1

[Qemu-devel] [PATCH v7 27/54] tests/tcg: move MIPS specific tests into subdir

[Alex Bennée]

These only need to be built for MIPS guests.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
 tests/tcg/README  | 11 ---
 tests/tcg/mips/README |  7 +++
 tests/tcg/{ => mips}/hello-mips.c |  0
 3 files changed, 7 insertions(+), 11 deletions(-)
 create mode 100644 tests/tcg/mips/README
 rename tests/tcg/{ => mips}/hello-mips.c (100%)

diff --git a/tests/tcg/README b/tests/tcg/README
index 625f2326e6..a5643d33e7 100644
--- a/tests/tcg/README
+++ b/tests/tcg/README
@@ -3,17 +3,6 @@ regression testing. Tests are either multi-arch, meaning they 
can be
 built for all guest architectures that support linux-user executable,
 or they are architecture specific.
 
-
-
-MIPS
-
-
-hello-mips
---
-
-hello-mipsel
-
-
 CRIS
 
 The testsuite for CRIS is in tests/tcg/cris.  You can run it
diff --git a/tests/tcg/mips/README b/tests/tcg/mips/README
new file mode 100644
index 00..e5bbc58ec5
--- /dev/null
+++ b/tests/tcg/mips/README
@@ -0,0 +1,7 @@
+MIPS
+
+
+hello-mips
+--
+
+A very simple inline assembly, write syscall based hello world
diff --git a/tests/tcg/hello-mips.c b/tests/tcg/mips/hello-mips.c
similarity index 100%
rename from tests/tcg/hello-mips.c
rename to tests/tcg/mips/hello-mips.c
-- 
2.17.1

[Qemu-devel] [PATCH v7 36/54] tests/tcg: enable building for m68k

[Alex Bennée]

As before, using Debian SID compilers.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 

---
v5
  - add EXTRA_RUNS for mmap tests
---
 tests/docker/Makefile.include |  1 +
 tests/docker/dockerfiles/debian-m68k-cross.docker | 12 
 tests/tcg/m68k/Makefile.include   |  2 ++
 tests/tcg/m68k/Makefile.target|  7 +++
 4 files changed, 22 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-m68k-cross.docker
 create mode 100644 tests/tcg/m68k/Makefile.include
 create mode 100644 tests/tcg/m68k/Makefile.target

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 65e3dee741..59dd8e314d 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -72,6 +72,7 @@ docker-image-debian-sid: NOCACHE=1
 
 docker-image-debian-alpha-cross: docker-image-debian-sid
 docker-image-debian-hppa-cross: docker-image-debian-sid
+docker-image-debian-m68k-cross: docker-image-debian-sid
 docker-image-travis: NOUSER=1
 
 # Specialist build images, sometimes very limited tools
diff --git a/tests/docker/dockerfiles/debian-m68k-cross.docker 
b/tests/docker/dockerfiles/debian-m68k-cross.docker
new file mode 100644
index 00..21ba3b0132
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-m68k-cross.docker
@@ -0,0 +1,12 @@
+#
+# Docker cross-compiler target
+#
+# This docker target builds on the debian sid base image which
+# contains cross compilers for Debian "ports" targets.
+#
+FROM qemu:debian-sid
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+apt-get install -y --no-install-recommends \
+gcc-m68k-linux-gnu \
+libc6-dev-m68k-cross
diff --git a/tests/tcg/m68k/Makefile.include b/tests/tcg/m68k/Makefile.include
new file mode 100644
index 00..cd7c6bf50d
--- /dev/null
+++ b/tests/tcg/m68k/Makefile.include
@@ -0,0 +1,2 @@
+DOCKER_IMAGE=debian-m68k-cross
+DOCKER_CROSS_COMPILER=m68k-linux-gnu-gcc
diff --git a/tests/tcg/m68k/Makefile.target b/tests/tcg/m68k/Makefile.target
new file mode 100644
index 00..62f109eef4
--- /dev/null
+++ b/tests/tcg/m68k/Makefile.target
@@ -0,0 +1,7 @@
+# -*- Mode: makefile -*-
+#
+# m68k specific tweaks - specifically masking out broken tests
+#
+
+# On m68k Linux supports 4k and 8k pages (but 8k is currently broken)
+EXTRA_RUNS+=run-test-mmap-4096 # run-test-mmap-8192
-- 
2.17.1

[Qemu-devel] [PATCH v7 50/54] docker: docker.py add check sub-command

[Alex Bennée]

This command allows you to check if we need to re-build a docker
image. If the image isn't in the repository or the checksums don't
match then we return false and some text (for processing in
makefiles).

Signed-off-by: Alex Bennée 

---
  - fix prints to be "modern"
  - PEP8 fixes
---
 tests/docker/docker.py | 25 +
 1 file changed, 25 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 2593a12d04..3666197a16 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -442,6 +442,31 @@ class CcCommand(SubCommand):
 return Docker().command("run", cmd, args.quiet)
 
 
+class CheckCommand(SubCommand):
+"""Check if we need to re-build a docker image out of a dockerfile.
+Arguments:  """
+name = "check"
+
+def args(self, parser):
+parser.add_argument("tag",
+help="Image Tag")
+parser.add_argument("dockerfile",
+help="Dockerfile name")
+
+def run(self, args, argv):
+dockerfile = open(args.dockerfile, "rb").read()
+tag = args.tag
+
+dkr = Docker()
+if dkr.image_matches_dockerfile(tag, dockerfile):
+if not args.quiet:
+print("Image is up to date.")
+return 0
+else:
+print("Image needs updating")
+return 1
+
+
 def main():
 parser = argparse.ArgumentParser(description="A Docker helper",
 usage="%s  ..." % os.path.basename(sys.argv[0]))
-- 
2.17.1

[Qemu-devel] [PATCH v7 46/54] tests/tcg: add run, diff, and skip helper macros

[Alex Bennée]

As we aren't using the default runners for all the test cases it is
easy to miss out things like timeouts. To help with this we add some
helpers and use them so we only need to make core changes in one
place.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v5
  - new for v5
v5.1
  - diff-test->diff-out
v7
  - mention $(TARGET_NAME) when skipping
---
 tests/tcg/Makefile  | 13 ++---
 tests/tcg/aarch64/Makefile.target   |  6 ++
 tests/tcg/arm/Makefile.target   |  6 ++
 tests/tcg/i386/Makefile.target  | 10 --
 tests/tcg/multiarch/Makefile.target |  4 ++--
 5 files changed, 20 insertions(+), 19 deletions(-)

diff --git a/tests/tcg/Makefile b/tests/tcg/Makefile
index e7dbcdb5bf..bf06415390 100644
--- a/tests/tcg/Makefile
+++ b/tests/tcg/Makefile
@@ -34,6 +34,15 @@
 
 quiet-command = $(if $(V),$1,$(if $(2),@printf "  %-7s %s\n" $2 $3 && $1, @$1))
 
+# $1 = test name, $2 = cmd, $3 = desc
+run-test = $(call quiet-command, timeout $(TIMEOUT) $2 > $1.out,"TEST",$3)
+
+# $1 = test name, $2 = reference
+diff-out = $(call quiet-command, diff -u $1.out $2 | head -n 10,"DIFF","$1.out 
with $2")
+
+# $1 = test name, $2 = reason
+skip-test = @printf "  SKIPPED %s on $(TARGET_NAME) because %s\n" $1 $2
+
 # Tests we are building
 TESTS=
 
@@ -84,9 +93,7 @@ RUN_TESTS=$(patsubst %,run-%, $(TESTS))
 RUN_TESTS+=$(EXTRA_RUNS)
 
 run-%: %
-   $(call quiet-command, \
-   timeout $(TIMEOUT) $(QEMU) $< > $<.out, \
-   "TEST", "$< on $(TARGET_NAME)")
+   $(call run-test, $<, $(QEMU) $<, "$< on $(TARGET_NAME)")
 
 .PHONY: run
 run: $(RUN_TESTS)
diff --git a/tests/tcg/aarch64/Makefile.target 
b/tests/tcg/aarch64/Makefile.target
index 7dba32138d..08c45b8470 100644
--- a/tests/tcg/aarch64/Makefile.target
+++ b/tests/tcg/aarch64/Makefile.target
@@ -13,7 +13,5 @@ TESTS:=$(AARCH64_TESTS)
 fcvt: LDFLAGS+=-lm
 
 run-fcvt: fcvt
-   $(call quiet-command, \
-   $(QEMU) $< > fcvt.out && \
-   diff -u $(AARCH64_SRC)/fcvt.ref fcvt.out, \
-   "TEST", "$< (default) on $(TARGET_NAME)")
+   $(call run-test,$<,$(QEMU) $<, "$< on $(TARGET_NAME)")
+   $(call diff-out,$<,$(AARCH64_SRC)/fcvt.ref)
diff --git a/tests/tcg/arm/Makefile.target b/tests/tcg/arm/Makefile.target
index 14b5435fc6..aa4e4e3782 100644
--- a/tests/tcg/arm/Makefile.target
+++ b/tests/tcg/arm/Makefile.target
@@ -24,10 +24,8 @@ fcvt: LDFLAGS+=-lm
 # fcvt: CFLAGS+=-march=armv8.2-a+fp16 -mfpu=neon-fp-armv8
 
 run-fcvt: fcvt
-   $(call quiet-command, \
-   $(QEMU) $< > fcvt.out && \
-   diff -u $(ARM_SRC)/fcvt.ref fcvt.out, \
-   "TEST", "$< (default) on $(TARGET_NAME)")
+   $(call run-test,fcvt,$(QEMU) $<,"$< on $(TARGET_NAME)")
+   $(call diff-out,fcvt,$(ARM_SRC)/fcvt.ref)
 endif
 
 # On ARM Linux only supports 4k pages
diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target
index cd173363ee..97b7c23cf1 100644
--- a/tests/tcg/i386/Makefile.target
+++ b/tests/tcg/i386/Makefile.target
@@ -31,7 +31,7 @@ test-i386: test-i386.c test-i386-code16.S test-i386-vm86.S 
test-i386.h test-i386
 
 # Specialist test runners
 run-runcom: runcom pi_10.com
-   $(call quiet-command, $(QEMU) ./runcom $(I386_SRC)/pi_10.com > 
runcom.out, "TEST", "$< on $(TARGET_NAME)")
+   $(call run-test,$<,$(QEMU) ./runcom $(I386_SRC)/pi_10.com,"$< on 
$(TARGET_NAME)")
 
 ifeq ($(SPEED), slow)
 
@@ -40,13 +40,11 @@ test-i386-fprem.ref: test-i386-fprem
 
 run-test-i386-fprem: TIMEOUT=60
 run-test-i386-fprem: test-i386-fprem
-   $(call quiet-command, \
-   $(QEMU) $< > $<.out && \
-   diff -u $(I386_SRC)/$<.ref $<.out, \
-   "TEST", "$< (default) on $(TARGET_NAME)")
+   $(call run-test,test-i386-fprem, $(QEMU) $<,"$< on $(TARGET_NAME)")
+   $(call diff-out,test-i386-fprem, $(I386_SRC)/$<.ref)
 else
 run-test-i386-fprem: test-i386-fprem
-   $(call quiet-command, /bin/true, "SLOW TEST", "$< SKIPPED on 
$(TARGET_NAME)")
+   $(call skip-test, $<, "SLOW")
 endif
 
 # On i386 and x86_64 Linux only supports 4k pages (large pages are a different 
hack)
diff --git a/tests/tcg/multiarch/Makefile.target 
b/tests/tcg/multiarch/Makefile.target
index 90e45a881e..b77084c146 100644
--- a/tests/tcg/multiarch/Makefile.target
+++ b/tests/tcg/multiarch/Makefile.target
@@ -27,10 +27,10 @@ testthread: LDFLAGS+=-lpthread
 
 # default case (host page size)
 run-test-mmap: test-mmap
-   $(call quiet-command, $(QEMU) $< > test-mmap.out, "TEST", \
+   $(call run-test, test-mmap, $(QEMU) $<, \
"$< (default) on $(TARGET_NAME)")
 
 # additional page sizes (defined by each architecture adding to EXTRA_RUNS)
 run-test-mmap-%: test-mmap
-   $(call quiet-command, $(QEMU) -p $* $< > test-mmap-$*.out, "TEST", \
+   $(call run-test, test-mmap-$*, $(QEMU) -p $* $<,\
"$< ($* byte pages) on $(TARGET_NAME)")
-- 

[Qemu-devel] [PATCH v7 13/54] tests/tcg/multiarch: move most output to stdout

[Alex Bennée]

The default test run outputs to stdout so it can be re-directed.
Errors are still reported to stderr.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: Richard Henderson 

---
v4
  - remove space in fprintf () to keep checkpatch happy
---
 tests/tcg/multiarch/test-mmap.c | 40 -
 1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/tests/tcg/multiarch/test-mmap.c b/tests/tcg/multiarch/test-mmap.c
index cdefadfa4c..5c0afe6e49 100644
--- a/tests/tcg/multiarch/test-mmap.c
+++ b/tests/tcg/multiarch/test-mmap.c
@@ -36,7 +36,7 @@
 do \
 {  \
   if (!(x)) {  \
-fprintf (stderr, "FAILED at %s:%d\n", __FILE__, __LINE__); \
+fprintf(stderr, "FAILED at %s:%d\n", __FILE__, __LINE__); \
 exit (EXIT_FAILURE);   \
   }\
 } while (0)
@@ -57,7 +57,7 @@ void check_aligned_anonymous_unfixed_mmaps(void)
uintptr_t p;
int i;
 
-   fprintf (stderr, "%s", __func__);
+   fprintf(stdout, "%s", __func__);
for (i = 0; i < 0x1fff; i++)
{
size_t len;
@@ -106,7 +106,7 @@ void check_aligned_anonymous_unfixed_mmaps(void)
munmap (p4, len);
munmap (p5, len);
}
-   fprintf (stderr, " passed\n");
+   fprintf(stdout, " passed\n");
 }
 
 void check_large_anonymous_unfixed_mmap(void)
@@ -115,7 +115,7 @@ void check_large_anonymous_unfixed_mmap(void)
uintptr_t p;
size_t len;
 
-   fprintf (stderr, "%s", __func__);
+   fprintf(stdout, "%s", __func__);
 
len = 0x0200;
p1 = mmap(NULL, len, PROT_READ, 
@@ -130,7 +130,7 @@ void check_large_anonymous_unfixed_mmap(void)
/* Make sure we can read from the entire area.  */
memcpy (dummybuf, p1, pagesize);
munmap (p1, len);
-   fprintf (stderr, " passed\n");
+   fprintf(stdout, " passed\n");
 }
 
 void check_aligned_anonymous_unfixed_colliding_mmaps(void)
@@ -141,7 +141,7 @@ void check_aligned_anonymous_unfixed_colliding_mmaps(void)
uintptr_t p;
int i;
 
-   fprintf (stderr, "%s", __func__);
+   fprintf(stdout, "%s", __func__);
for (i = 0; i < 0x2fff; i++)
{
int nlen;
@@ -180,7 +180,7 @@ void check_aligned_anonymous_unfixed_colliding_mmaps(void)
munmap (p2, pagesize);
munmap (p3, nlen);
}
-   fprintf (stderr, " passed\n");
+   fprintf(stdout, " passed\n");
 }
 
 void check_aligned_anonymous_fixed_mmaps(void)
@@ -194,7 +194,7 @@ void check_aligned_anonymous_fixed_mmaps(void)
addr = mmap(NULL, pagesize * 40, PROT_READ | PROT_WRITE, 
MAP_PRIVATE | MAP_ANONYMOUS,
-1, 0);
-   fprintf (stderr, "%s addr=%p", __func__, addr);
+   fprintf(stdout, "%s addr=%p", __func__, addr);
fail_unless (addr != MAP_FAILED);
 
for (i = 0; i < 40; i++)
@@ -212,7 +212,7 @@ void check_aligned_anonymous_fixed_mmaps(void)
munmap (p1, pagesize);
addr += pagesize;
}
-   fprintf (stderr, " passed\n");
+   fprintf(stdout, " passed\n");
 }
 
 void check_aligned_anonymous_fixed_mmaps_collide_with_host(void)
@@ -225,8 +225,8 @@ void 
check_aligned_anonymous_fixed_mmaps_collide_with_host(void)
/* Find a suitable address to start with.  Right were the x86 hosts
 stack is.  */
addr = ((void *)0x8000);
-   fprintf (stderr, "%s addr=%p", __func__, addr);
-   fprintf (stderr, "FIXME: QEMU fails to track pages used by the host.");
+   fprintf(stdout, "%s addr=%p", __func__, addr);
+   fprintf(stdout, "FIXME: QEMU fails to track pages used by the host.");
 
for (i = 0; i < 20; i++)
{
@@ -243,7 +243,7 @@ void 
check_aligned_anonymous_fixed_mmaps_collide_with_host(void)
munmap (p1, pagesize);
addr += pagesize;
}
-   fprintf (stderr, " passed\n");
+   fprintf(stdout, " passed\n");
 }
 
 void check_file_unfixed_mmaps(void)
@@ -252,7 +252,7 @@ void check_file_unfixed_mmaps(void)
uintptr_t p;
int i;
 
-   fprintf (stderr, "%s", __func__);
+   fprintf(stdout, "%s", __func__);
for (i = 0; i < 0x10; i++)
{
size_t len;
@@ -294,7 +294,7 @@ void check_file_unfixed_mmaps(void)
munmap (p2, len);
munmap (p3, len);
}
-   fprintf (stderr, " passed\n");
+   fprintf(stdout, " passed\n");
 }
 
 void check_file_unfixed_eof_mmaps(void)
@@ -304,7 +304,7 @@ void check_file_unfixed_eof_mmaps(void)
uintptr_t p;
int i;
 
-   fprintf (stderr, "%s", __func__);
+   

[Qemu-devel] [PATCH v7 44/54] Makefile.target: add (clean-/build-)guest-tests targets

[Alex Bennée]

Now all the build infrastructure is in place we can build tests for
each guest that we support. That support mainly depends on having
cross compilers installed or docker setup. To keep all the logic for
that together we put the rules in tests/tcg/Makefile.include and
include it from the main Makefile.target.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v4
  - pulled Makefile.include bits from next patch
  - added qemu-$(TARGET_NAME) dependency for running tests
  - accept SPEED parameter to run slow tests
  - pass BUILD_STATIC
  - update MAINTAINERS
v6
  - bare make -> $(MAKE) for sub-makes
v7
  - fixed spelling
  - shorten CROSS-BUILD to BUILD and RUN-TESTS to RUN with additional
info to the right.
---
 MAINTAINERS|  1 +
 Makefile.target|  5 +++
 tests/tcg/Makefile.include | 88 ++
 3 files changed, 94 insertions(+)
 create mode 100644 tests/tcg/Makefile.include

diff --git a/MAINTAINERS b/MAINTAINERS
index 3795a32c27..6a8d42ab3d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2153,6 +2153,7 @@ W: http://patchew.org/QEMU/
 Guest Test Compilation Support
 M: Alex Bennée 
 F: tests/tcg/Makefile
+F: tests/tcg/Makefile.include
 L: qemu-devel@nongnu.org
 
 Documentation
diff --git a/Makefile.target b/Makefile.target
index dad2cf8778..a9d8928f96 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -36,6 +36,11 @@ endif
 PROGS=$(QEMU_PROG) $(QEMU_PROGW)
 STPFILES=
 
+# Makefile Tests
+ifdef CONFIG_USER_ONLY
+include $(SRC_PATH)/tests/tcg/Makefile.include
+endif
+
 config-target.h: config-target.h-timestamp
 config-target.h-timestamp: config-target.mak
 
diff --git a/tests/tcg/Makefile.include b/tests/tcg/Makefile.include
new file mode 100644
index 00..57470b2a2c
--- /dev/null
+++ b/tests/tcg/Makefile.include
@@ -0,0 +1,88 @@
+# -*- Mode: makefile -*-
+#
+# TCG tests (per-target rules)
+#
+# This Makefile fragement is included from the per-target
+# Makefile.target so will be invoked for each linux-user program we
+# build. We have two options for compiling, either using a configured
+# guest compiler or calling one of our docker images to do it for us.
+#
+
+# The per ARCH makefile, if it exists, holds extra information about
+# useful docker images or alternative compiler flags.
+
+-include $(SRC_PATH)/tests/tcg/$(TARGET_BASE_ARCH)/Makefile.include
+-include $(SRC_PATH)/tests/tcg/$(TARGET_NAME)/Makefile.include
+
+GUEST_BUILD=
+TCG_MAKE=$(SRC_PATH)/tests/tcg/Makefile
+# Support installed Cross Compilers
+
+ifdef CROSS_CC_GUEST
+
+.PHONY: cross-build-guest-tests
+cross-build-guest-tests:
+   $(call quiet-command, \
+  (mkdir -p tests && cd tests && \
+   $(MAKE) -f $(TCG_MAKE) CC=$(CROSS_CC_GUEST) \
+   BUILD_STATIC=$(CROSS_CC_GUEST_STATIC) \
+   EXTRA_CFLAGS=$(CROSS_CC_GUEST_CFLAGS)), \
+   "BUILD","$(TARGET_NAME) guest-tests with $(CROSS_CC_GUEST)")
+
+GUEST_BUILD=cross-build-guest-tests
+
+endif
+
+# Support building with Docker
+
+ifeq ($(HAVE_USER_DOCKER)$(GUEST_BUILD),y)
+ifneq ($(DOCKER_IMAGE),)
+
+# We also need the Docker make rules to depend on
+include $(SRC_PATH)/tests/docker/Makefile.include
+
+DOCKER_COMPILE_CMD="$(DOCKER_SCRIPT) cc --user $(shell id -u) \
+   --cc $(DOCKER_CROSS_COMPILER) \
+   -i qemu:$(DOCKER_IMAGE) \
+   -s $(SRC_PATH) -- "
+DOCKER_PREREQ=docker-image-$(DOCKER_IMAGE)
+
+.PHONY: docker-build-guest-tests
+docker-build-guest-tests: $(DOCKER_PREREQ)
+   $(call quiet-command, \
+ (mkdir -p tests && cd tests && \
+  $(MAKE) -f $(TCG_MAKE) CC=$(DOCKER_COMPILE_CMD) \
+   BUILD_STATIC=y \
+   EXTRA_CFLAGS=$(DOCKER_CROSS_COMPILER_CFLAGS)), \
+   "BUILD","$(TARGET_NAME) guest-tests with docker qemu:$(DOCKER_IMAGE)")
+
+GUEST_BUILD=docker-build-guest-tests
+
+endif
+endif
+
+# Final targets
+.PHONY: guest-tests
+
+ifneq ($(GUEST_BUILD),)
+guest-tests: $(GUEST_BUILD)
+
+run-guest-tests: guest-tests qemu-$(TARGET_NAME)
+   $(call quiet-command, \
+   (cd tests && $(MAKE) -f $(TCG_MAKE) SPEED=$(SPEED) run), \
+   "RUN", "tests for $(TARGET_NAME)")
+
+else
+guest-tests:
+   $(call quiet-command, /bin/true, "BUILD", \
+   "$(TARGET_NAME) guest-tests SKIPPED")
+
+run-guest-tests:
+   $(call quiet-command, /bin/true, "RUN", \
+   "tests for $(TARGET_NAME) SKIPPED")
+endif
+
+# It doesn't matter if these don't exits
+.PHONY: clean-guest-tests
+clean-guest-tests:
+   rm -rf tests || echo "no $(TARGET_NAME) tests to remove"
-- 
2.17.1

[Qemu-devel] [PATCH v7 34/54] tests/tcg/alpha: add Alpha specific tests

[Alex Bennée]

These tests did use their own crt.o stub however that is a little
stone age so we drop crt.S and just statically link to the cross
compilers libraries.

Signed-off-by: Alex Bennée 
Reviewed-by: Richard Henderson 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v4
  - drop crt.s
  - drop skipping of threadtest
  - drop write() hack, include unistd.h
v5
  - merge fixes
---
 tests/tcg/alpha/Makefile| 35 -
 tests/tcg/alpha/Makefile.target | 12 +++
 tests/tcg/alpha/crt.s   | 26 
 tests/tcg/alpha/hello-alpha.c   |  2 ++
 tests/tcg/alpha/test-cond.c |  1 +
 tests/tcg/alpha/test-ovf.c  |  2 ++
 6 files changed, 17 insertions(+), 61 deletions(-)
 delete mode 100644 tests/tcg/alpha/Makefile
 delete mode 100644 tests/tcg/alpha/crt.s

diff --git a/tests/tcg/alpha/Makefile b/tests/tcg/alpha/Makefile
deleted file mode 100644
index 2b1f03d048..00
--- a/tests/tcg/alpha/Makefile
+++ /dev/null
@@ -1,35 +0,0 @@
-CROSS=alpha-linux-gnu-
-CC=$(CROSS)gcc
-AS=$(CROSS)as
-
-SIM=../../alpha-linux-user/qemu-alpha
-
-CFLAGS=-O
-LINK=$(CC) -o $@ crt.o $< -nostdlib
-
-TESTS=test-cond test-cmov
-
-all: hello-alpha $(TESTS)
-
-hello-alpha: hello-alpha.o crt.o
-   $(LINK)
-
-test-cond: test-cond.o crt.o
-   $(LINK)
-
-test-cmov.o: test-cond.c
-   $(CC) -c $(CFLAGS) -DTEST_CMOV -o $@ $<
-
-test-cmov: test-cmov.o crt.o
-   $(LINK)
-
-test-ovf: test-ovf.o crt.o
-   $(LINK)
-
-check: $(TESTS)
-   for f in $(TESTS); do $(SIM) $$f || exit 1; done
-
-clean:
-   $(RM) *.o *~ hello-alpha $(TESTS)
-
-.PHONY: clean all check
diff --git a/tests/tcg/alpha/Makefile.target b/tests/tcg/alpha/Makefile.target
index bedf1b6444..a585080328 100644
--- a/tests/tcg/alpha/Makefile.target
+++ b/tests/tcg/alpha/Makefile.target
@@ -2,5 +2,17 @@
 #
 # Alpha specific tweaks
 
+ALPHA_SRC=$(SRC_PATH)/tests/tcg/alpha
+VPATH+=$(ALPHA_SRC)
+
+ALPHA_TESTS=hello-alpha test-cond test-cmov test-ovf
+TESTS+=$(ALPHA_TESTS)
+
+test-cmov: EXTRA_CFLAGS=-DTEST_CMOV
+test-cmov: test-cond.c
+   $(CC) $(CFLAGS) $(EXTRA_CFLAGS) $< -o $@ $(LDFLAGS)
+
+run-test-cmov: test-cmov
+
 # On Alpha Linux only supports 8k pages
 EXTRA_RUNS+=run-test-mmap-8192
diff --git a/tests/tcg/alpha/crt.s b/tests/tcg/alpha/crt.s
deleted file mode 100644
index 31af8825bc..00
--- a/tests/tcg/alpha/crt.s
+++ /dev/null
@@ -1,26 +0,0 @@
-   .text
-
-   .globl _start
-   .ent _start,0
-_start:
-   .frame $15,0,$15
-   br $29,1f
-1: ldgp $29, 0($29)
-   .prologue 0
-   ldq $27,main($29) !literal!1
-   jsr $26,($27)
-   or $0,$0,$16
-   .end _start
-
-   .globl _exit
-_exit:
-   lda $0,1
-   callsys
-
-   call_pal 0
-
-   .globl write
-write:
-   lda $0,4
-   callsys
-   ret
diff --git a/tests/tcg/alpha/hello-alpha.c b/tests/tcg/alpha/hello-alpha.c
index 79892e6522..84e43b2fc4 100644
--- a/tests/tcg/alpha/hello-alpha.c
+++ b/tests/tcg/alpha/hello-alpha.c
@@ -1,3 +1,5 @@
+#include 
+
 int main (void)
 {
   write (1, "hello\n", 6);
diff --git a/tests/tcg/alpha/test-cond.c b/tests/tcg/alpha/test-cond.c
index 74adffaa69..e625313b3e 100644
--- a/tests/tcg/alpha/test-cond.c
+++ b/tests/tcg/alpha/test-cond.c
@@ -1,3 +1,4 @@
+#include 
 
 #ifdef TEST_CMOV
 
diff --git a/tests/tcg/alpha/test-ovf.c b/tests/tcg/alpha/test-ovf.c
index 01c80e7525..17892f1e89 100644
--- a/tests/tcg/alpha/test-ovf.c
+++ b/tests/tcg/alpha/test-ovf.c
@@ -1,3 +1,5 @@
+#include 
+
 static long test_subqv (long a, long b)
 {
   long res;
-- 
2.17.1

[Qemu-devel] [PATCH v7 18/54] tests/tcg/i386: add runner for test-i386-fprem

[Alex Bennée]

The runner needs to compare against a reference run. We also only run
this test when SPEED=slow as it takes a while.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v5
  - include test-i386-fprem.ref
v5.1
  - remove the 69Mb test-i386-fprem.ref
  - generate test-i386-fprem.ref on developers machine
  - add TIMEOUT=60
---
 tests/tcg/i386/Makefile.target   | 16 
 tests/tcg/i386/test-i386-fprem.c | 12 +---
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target
index 8b46cc021e..64d241cfdf 100644
--- a/tests/tcg/i386/Makefile.target
+++ b/tests/tcg/i386/Makefile.target
@@ -33,5 +33,21 @@ test-i386: test-i386.c test-i386-code16.S test-i386-vm86.S 
test-i386.h test-i386
 run-runcom: runcom pi_10.com
$(call quiet-command, $(QEMU) ./runcom $(I386_SRC)/pi_10.com > 
runcom.out, "TEST", "$< on $(TARGET_NAME)")
 
+ifeq ($(SPEED), slow)
+
+test-i386-fprem.ref: test-i386-fprem
+   $(call quiet-command, ./$< > $@,"GENREF","generating $@")
+
+run-test-i386-fprem: TIMEOUT=60
+run-test-i386-fprem: test-i386-fprem
+   $(call quiet-command, \
+   $(QEMU) $< > $<.out && \
+   diff -u $(I386_SRC)/$<.ref $<.out, \
+   "TEST", "$< (default) on $(TARGET_NAME)")
+else
+run-test-i386-fprem: test-i386-fprem
+   $(call quiet-command, /bin/true, "SLOW TEST", "$< SKIPPED on 
$(TARGET_NAME)")
+endif
+
 # On i386 and x86_64 Linux only supports 4k pages (large pages are a different 
hack)
 EXTRA_RUNS+=run-test-mmap-4096
diff --git a/tests/tcg/i386/test-i386-fprem.c b/tests/tcg/i386/test-i386-fprem.c
index 1a71623204..66f5a9657d 100644
--- a/tests/tcg/i386/test-i386-fprem.c
+++ b/tests/tcg/i386/test-i386-fprem.c
@@ -23,7 +23,10 @@
  *  along with this program; if not, see .
  */
 
-#include "qemu/osdep.h"
+#include 
+#include 
+
+#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
 
 /*
  * Inspired by 's union ieee854_long_double, but with single
@@ -39,7 +42,7 @@ union float80u {
 unsigned int exponent:15;
 unsigned int negative:1;
 unsigned int empty:16;
-} QEMU_PACKED ieee;
+} __attribute__((packed)) ieee;
 
 /* This is for NaNs in the IEEE 854 double-extended-precision format.  */
 struct {
@@ -49,7 +52,7 @@ union float80u {
 unsigned int exponent:15;
 unsigned int negative:1;
 unsigned int empty:16;
-} QEMU_PACKED ieee_nan;
+} __attribute__((packed)) ieee_nan;
 };
 
 #define IEEE854_LONG_DOUBLE_BIAS 0x3fff
@@ -229,6 +232,7 @@ static void test_fprem_cases(void)
 do_fprem_stack_underflow();
 
 printf("= invalid operation =\n");
+do_fprem(q_nan.d, 1.0);
 do_fprem(s_nan.d, 1.0);
 do_fprem(1.0, 0.0);
 do_fprem(pos_inf.d, 1.0);
@@ -238,6 +242,8 @@ static void test_fprem_cases(void)
 do_fprem(pos_denorm.d, 1.0);
 do_fprem(1.0, pos_denorm.d);
 
+do_fprem(smallest_positive_norm.d, smallest_positive_norm.d);
+
 /* printf("= underflow =\n"); */
 /* TODO: Is there a case where FPREM raises underflow? */
 }
-- 
2.17.1

[Qemu-devel] [PATCH v7 24/54] tests/tcg/arm: fix up test-arm-iwmmxt test

[Alex Bennée]

We need to rename the source file to a .S so we can do a single-line
assemble and link invocation. We also specify the additional CFLAGS
for the compile as it's a non-standard ARM binary.

Signed-off-by: Alex Bennée 
[rth: force fpu configuration]
Signed-off-by: Richard Henderson 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: Richard Henderson 
---
v4
  - include rth's -mfpu=fpv4-sp-d16
v5
  - merge fixes
---
 tests/tcg/arm/Makefile.target  | 5 +
 tests/tcg/arm/{test-arm-iwmmxt.s => test-arm-iwmmxt.S} | 0
 2 files changed, 5 insertions(+)
 rename tests/tcg/arm/{test-arm-iwmmxt.s => test-arm-iwmmxt.S} (100%)

diff --git a/tests/tcg/arm/Makefile.target b/tests/tcg/arm/Makefile.target
index 0312293dca..cadb385890 100644
--- a/tests/tcg/arm/Makefile.target
+++ b/tests/tcg/arm/Makefile.target
@@ -8,9 +8,14 @@ ARM_SRC=$(SRC_PATH)/tests/tcg/arm
 # Set search path for all sources
 VPATH  += $(ARM_SRC)
 
+TESTS += hello-arm test-arm-iwmmxt
 
 hello-arm: CFLAGS+=-marm -ffreestanding
 hello-arm: LDFLAGS+=-nostdlib
 
+test-arm-iwmmxt: CFLAGS+=-marm -march=iwmmxt -mabi=aapcs -mfpu=fpv4-sp-d16
+test-arm-iwmmxt: test-arm-iwmmxt.S
+   $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
+
 # On ARM Linux only supports 4k pages
 EXTRA_RUNS+=run-test-mmap-4096
diff --git a/tests/tcg/arm/test-arm-iwmmxt.s b/tests/tcg/arm/test-arm-iwmmxt.S
similarity index 100%
rename from tests/tcg/arm/test-arm-iwmmxt.s
rename to tests/tcg/arm/test-arm-iwmmxt.S
-- 
2.17.1

[Qemu-devel] [PATCH v7 14/54] tests/tcg: move i386 specific tests into subdir

[Alex Bennée]

These only need to be built for i386 guests. This includes a stub
tests/tcg/i386/Makfile.target which absorbs some of what was in
tests/tcg/Makefile.

Signed-off-by: Alex Bennée 
Reviewed-by: Thomas Huth 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: Richard Henderson 

---
v2
  - move VPATH and TESTs setup into i386/Makefile.target
  - set CFLAGS+=-m32 for cross building
v4
  - update MAINATINERS
---
 MAINTAINERS |   1 +
 tests/tcg/README|  39 
 tests/tcg/i386/Makefile.target  |  30 ++
 tests/tcg/i386/README   |  38 +++
 tests/tcg/{ => i386}/hello-i386.c   |   0
 tests/tcg/{ => i386}/pi_10.com  | Bin
 tests/tcg/{ => i386}/runcom.c   |   0
 tests/tcg/{ => i386}/test-i386-code16.S |   0
 tests/tcg/{ => i386}/test-i386-fprem.c  |   0
 tests/tcg/{ => i386}/test-i386-muldiv.h |   0
 tests/tcg/{ => i386}/test-i386-shift.h  |   0
 tests/tcg/{ => i386}/test-i386-ssse3.c  |   0
 tests/tcg/{ => i386}/test-i386-vm86.S   |   0
 tests/tcg/{ => i386}/test-i386.c|   0
 tests/tcg/{ => i386}/test-i386.h|   0
 15 files changed, 69 insertions(+), 39 deletions(-)
 create mode 100644 tests/tcg/i386/Makefile.target
 create mode 100644 tests/tcg/i386/README
 rename tests/tcg/{ => i386}/hello-i386.c (100%)
 rename tests/tcg/{ => i386}/pi_10.com (100%)
 rename tests/tcg/{ => i386}/runcom.c (100%)
 rename tests/tcg/{ => i386}/test-i386-code16.S (100%)
 rename tests/tcg/{ => i386}/test-i386-fprem.c (100%)
 rename tests/tcg/{ => i386}/test-i386-muldiv.h (100%)
 rename tests/tcg/{ => i386}/test-i386-shift.h (100%)
 rename tests/tcg/{ => i386}/test-i386-ssse3.c (100%)
 rename tests/tcg/{ => i386}/test-i386-vm86.S (100%)
 rename tests/tcg/{ => i386}/test-i386.c (100%)
 rename tests/tcg/{ => i386}/test-i386.h (100%)

diff --git a/MAINTAINERS b/MAINTAINERS
index eccd81e8e7..ea298cc910 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -283,6 +283,7 @@ M: Richard Henderson 
 M: Eduardo Habkost 
 S: Maintained
 F: target/i386/
+F: tests/tcg/i386/
 F: hw/i386/
 F: disas/i386.c
 T: git git://github.com/ehabkost/qemu.git x86-next
diff --git a/tests/tcg/README b/tests/tcg/README
index 0890044cf0..469504c4cb 100644
--- a/tests/tcg/README
+++ b/tests/tcg/README
@@ -3,45 +3,6 @@ regression testing. Tests are either multi-arch, meaning they 
can be
 built for all guest architectures that support linux-user executable,
 or they are architecture specific.
 
-i386
-
-
-test-i386
--
-
-This program executes most of the 16 bit and 32 bit x86 instructions and
-generates a text output, for comparison with the output obtained with
-a real CPU or another emulator.
-
-The Linux system call modify_ldt() is used to create x86 selectors
-to test some 16 bit addressing and 32 bit with segmentation cases.
-
-The Linux system call vm86() is used to test vm86 emulation.
-
-Various exceptions are raised to test most of the x86 user space
-exception reporting.
-
-linux-test
---
-
-This program tests various Linux system calls. It is used to verify
-that the system call parameters are correctly converted between target
-and host CPUs.
-
-test-i386-fprem

-
-runcom
---
-
-test-mmap
--
-
-sha1
-
-
-hello-i386
---
 
 
 ARM
diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target
new file mode 100644
index 00..2f27b65e2d
--- /dev/null
+++ b/tests/tcg/i386/Makefile.target
@@ -0,0 +1,30 @@
+# i386 cross compile notes
+
+I386_SRC=$(SRC_PATH)/tests/tcg/i386
+
+# Set search path for all sources
+VPATH  += $(I386_SRC)
+
+I386_SRCS=$(notdir $(wildcard $(I386_SRC)/*.c))
+I386_TESTS=$(I386_SRCS:.c=)
+
+# Update TESTS
+TESTS+=$(I386_TESTS)
+
+ifneq ($(TARGET_NAME),x86_64)
+CFLAGS+=-m32
+endif
+
+#
+# hello-i386 is a barebones app
+#
+hello-i386: CFLAGS+=-ffreestanding
+hello-i386: LDFLAGS+=-nostdlib
+
+#
+# test-386 includes a couple of additional objects that need to be linked 
together
+#
+
+test-i386: test-i386.c test-i386-code16.S test-i386-vm86.S test-i386.h 
test-i386-shift.h test-i386-muldiv.h
+   $(CC) $(CFLAGS) $(LDFLAGS) -o $@ \
+  $(

[Qemu-devel] [PATCH v7 45/54] tests/Makefile.include: add [build|clean|check]-tcg targets

[Alex Bennée]

This will ensure all linux-user targets build their guest test
programs and ensure check-tcg will run the respective tests.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v2
  - use -include instead of complex macro stuff
  - also include TARGET_BASE_ARCH/Makefile
v3
  - add build-tcg, make check-tcg actually run tests
---
 tests/Makefile.include | 26 +-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index ca91da26cb..c8faa01a33 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -10,6 +10,7 @@ check-help:
@echo " $(MAKE) check-speed  Run qobject speed tests"
@echo " $(MAKE) check-qapi-schemaRun QAPI schema tests"
@echo " $(MAKE) check-block  Run block tests"
+   @echo " $(MAKE) check-tcgRun TCG tests"
@echo " $(MAKE) check-report.htmlGenerates an HTML test report"
@echo " $(MAKE) check-clean  Clean the tests"
@echo
@@ -930,6 +931,30 @@ check-report.xml: $(patsubst %,check-report-qtest-%.xml, 
$(QTEST_TARGETS)) check
 check-report.html: check-report.xml
$(call quiet-command,gtester-report $< > $@,"GEN","$@")
 
+# Per guest TCG tests
+
+LINUX_USER_TARGETS=$(filter %-linux-user,$(TARGET_LIST))
+BUILD_TCG_TARGET_RULES=$(patsubst %,build-tcg-tests-%, $(LINUX_USER_TARGETS))
+CLEAN_TCG_TARGET_RULES=$(patsubst %,clean-tcg-tests-%, $(LINUX_USER_TARGETS))
+RUN_TCG_TARGET_RULES=$(patsubst %,run-tcg-tests-%, $(LINUX_USER_TARGETS))
+
+build-tcg-tests-%:
+   $(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" 
TARGET_DIR="$*/" guest-tests,)
+
+run-tcg-tests-%: build-tcg-tests-%
+   $(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" 
TARGET_DIR="$*/" run-guest-tests,)
+
+clean-tcg-tests-%:
+   $(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" 
TARGET_DIR="$*/" clean-guest-tests,)
+
+.PHONY: build-tcg
+build-tcg: $(BUILD_TCG_TARGET_RULES)
+
+.PHONY: check-tcg
+check-tcg: $(RUN_TCG_TARGET_RULES)
+
+.PHONY: clean-tcg
+clean-tcg: $(CLEAN_TCG_TARGET_RULES)
 
 # Other tests
 
@@ -972,7 +997,6 @@ check-speed: $(patsubst %,check-%, $(check-speed-y))
 check-block: $(patsubst %,check-%, $(check-block-y))
 check: check-qapi-schema check-unit check-qtest check-decodetree
 check-clean:
-   $(MAKE) -C tests/tcg clean
rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), 
$(check-qtest-$(target)-y)) $(check-qtest-generic-y))
rm -f tests/test-qapi-gen-timestamp
-- 
2.17.1

[Qemu-devel] [PATCH v7 35/54] tests/tcg: enable building for HPPA

[Alex Bennée]

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 

---
v5
  - add EXTRA_RUNS for mmap tests
---
 tests/docker/Makefile.include |  1 +
 tests/docker/dockerfiles/debian-hppa-cross.docker | 12 
 tests/tcg/hppa/Makefile.include   |  2 ++
 tests/tcg/hppa/Makefile.target|  6 ++
 4 files changed, 21 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-hppa-cross.docker
 create mode 100644 tests/tcg/hppa/Makefile.include
 create mode 100644 tests/tcg/hppa/Makefile.target

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 8c24f8807e..65e3dee741 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -71,6 +71,7 @@ docker-image-debian-win64-cross: docker-image-debian8-mxe
 docker-image-debian-sid: NOCACHE=1
 
 docker-image-debian-alpha-cross: docker-image-debian-sid
+docker-image-debian-hppa-cross: docker-image-debian-sid
 docker-image-travis: NOUSER=1
 
 # Specialist build images, sometimes very limited tools
diff --git a/tests/docker/dockerfiles/debian-hppa-cross.docker 
b/tests/docker/dockerfiles/debian-hppa-cross.docker
new file mode 100644
index 00..ad443defac
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-hppa-cross.docker
@@ -0,0 +1,12 @@
+#
+# Docker cross-compiler target
+#
+# This docker target builds on the debian sid base image which
+# contains cross compilers for Debian "ports" targets.
+#
+FROM qemu:debian-sid
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+apt-get install -y --no-install-recommends \
+gcc-hppa-linux-gnu \
+libc6-dev-hppa-cross
diff --git a/tests/tcg/hppa/Makefile.include b/tests/tcg/hppa/Makefile.include
new file mode 100644
index 00..da2353430e
--- /dev/null
+++ b/tests/tcg/hppa/Makefile.include
@@ -0,0 +1,2 @@
+DOCKER_IMAGE=debian-hppa-cross
+DOCKER_CROSS_COMPILER=hppa-linux-gnu-gcc
diff --git a/tests/tcg/hppa/Makefile.target b/tests/tcg/hppa/Makefile.target
new file mode 100644
index 00..8bf01966bd
--- /dev/null
+++ b/tests/tcg/hppa/Makefile.target
@@ -0,0 +1,6 @@
+# -*- Mode: makefile -*-
+#
+# HPPA specific tweaks - specifically masking out broken tests
+
+# On parisc Linux supports 4K/16K/64K (but currently only 4k works)
+EXTRA_RUNS+=run-test-mmap-4096 # run-test-mmap-16384 run-test-mmap-65536
-- 
2.17.1

[Qemu-devel] [PATCH v7 32/54] tests/tcg: enable building for ppc64

[Alex Bennée]

Currently this just enables building the multiarch tests.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
 tests/tcg/ppc64le/Makefile.include | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 tests/tcg/ppc64le/Makefile.include

diff --git a/tests/tcg/ppc64le/Makefile.include 
b/tests/tcg/ppc64le/Makefile.include
new file mode 100644
index 00..d71cfc9aa7
--- /dev/null
+++ b/tests/tcg/ppc64le/Makefile.include
@@ -0,0 +1,2 @@
+DOCKER_IMAGE=debian-ppc64el-cross
+DOCKER_CROSS_COMPILER=powerpc64le-linux-gnu-gcc
-- 
2.17.1

[Qemu-devel] [PATCH v7 47/54] tests/tcg: override runners for broken tests

[Alex Bennée]

To get a clean run of check-tcg these tests are currently skipped:

  - hello-mips for mips
  - linux-test for sparc

Signed-off-by: Alex Bennée 
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: Philippe Mathieu-Daudé 
---
v4
  - document broken tests in commit
  - temporarily demote fcvt while fixes percolate upstream
v5
  - reinstate fcvt
  - remove test-mmap overrides (done with EXTRA_RUNS now)
v6
  - move to after the skip-test helper
v7
  - rm personal path
---
 tests/tcg/mips/Makefile.target| 3 +++
 tests/tcg/sparc64/Makefile.target | 7 ++-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/tests/tcg/mips/Makefile.target b/tests/tcg/mips/Makefile.target
index 71f0c2dd53..086625f533 100644
--- a/tests/tcg/mips/Makefile.target
+++ b/tests/tcg/mips/Makefile.target
@@ -17,3 +17,6 @@ hello-mips: LDFLAGS+=-nostdlib
 
 # For MIPS32 and 64 we have a bunch of extra tests in sub-directories
 # however they are intended for system tests.
+
+run-hello-mips: hello-mips
+   $(call skip-test, $<, "BROKEN")
diff --git a/tests/tcg/sparc64/Makefile.target 
b/tests/tcg/sparc64/Makefile.target
index 408dace783..5bd7f90583 100644
--- a/tests/tcg/sparc64/Makefile.target
+++ b/tests/tcg/sparc64/Makefile.target
@@ -1,6 +1,11 @@
 # -*- Mode: makefile -*-
 #
-# sparc specific tweaks
+# sparc specific tweaks and masking out broken tests
+
+# different from the other hangs:
+# tests/tcg/multiarch/linux-test.c:264: Value too large for defined data type 
(ret=-1, errno=92/Value too large for defined data type)
+run-linux-test: linux-test
+   $(call skip-test, $<, "BROKEN")
 
 # On Sparc64 Linux support 8k pages
 EXTRA_RUNS+=run-test-mmap-8192
-- 
2.17.1

[Qemu-devel] [PATCH v7 51/54] tests/Makefile: call sub-makes with SKIP_DOCKER_BUILD=1

[Alex Bennée]

As we now ensure all the images we are going to use are built in the
top level make file lets not over complicate things by running the
full script again. We do run the check script just in case someone
deletes the docker image while we are running.

Signed-off-by: Alex Bennée 
---
 tests/Makefile.include|  8 ++--
 tests/docker/Makefile.include | 13 -
 2 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 2dd0700a88..dad20e1b9b 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -947,10 +947,14 @@ $(foreach PROBE_TARGET,$(TARGET_LIST),
\
$(eval build-tcg-tests-$(PROBE_TARGET): $(DOCKER_PREREQ
 
 build-tcg-tests-%:
-   $(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" 
TARGET_DIR="$*/" guest-tests,)
+   $(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" \
+   SKIP_DOCKER_BUILD=1 TARGET_DIR="$*/" guest-tests, \
+   "BUILD", "TCG tests for $*")
 
 run-tcg-tests-%: build-tcg-tests-%
-   $(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" 
TARGET_DIR="$*/" run-guest-tests,)
+   $(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" \
+   SKIP_DOCKER_BUILD=1 TARGET_DIR="$*/" run-guest-tests, \
+   "RUN", "TCG tests for $*")
 
 clean-tcg-tests-%:
$(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" 
TARGET_DIR="$*/" clean-guest-tests,)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 87f449264f..c0f09505bb 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -34,7 +34,15 @@ docker-qemu-src: $(DOCKER_SRC_COPY)
 
 docker-image: ${DOCKER_TARGETS}
 
-# General rule for building docker images
+# General rule for building docker images. If we are a sub-make
+# invoked with SKIP_DOCKER_BUILD we still check the image is upto date
+# though
+ifdef SKIP_DOCKER_BUILD
+docker-image-%: $(DOCKER_FILES_DIR)/%.docker
+   $(call quiet-command, \
+   $(DOCKER_SCRIPT) check --quiet qemu:$* $<, \
+   "CHECK", "$*")
+else
 docker-image-%: $(DOCKER_FILES_DIR)/%.docker
@if test "$@" = docker-image-debian-bootstrap -a -z "$(EXECUTABLE)"; 
then \
echo WARNING: EXECUTABLE is not set, debootstrap may fail. 2>&1 
; \
@@ -46,6 +54,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
"BUILD","$*")
+endif
 
 # Enforce dependencies for composite images
 docker-image-debian: docker-image-debian9
@@ -65,7 +74,9 @@ docker-image-debian-win64-cross: docker-image-debian8-mxe
 # Debian SID images - we are tracking a rolling distro so we want to
 # force a re-build of the base image if we ever need to build one of
 # its children.
+ifndef SKIP_DOCKER_BUILD
 docker-image-debian-sid: NOCACHE=1
+endif
 
 docker-image-debian-alpha-cross: docker-image-debian-sid
 docker-image-debian-hppa-cross: docker-image-debian-sid
-- 
2.17.1

[Qemu-devel] [PATCH v7 17/54] tests/tcg/i386: fix test-i386

[Alex Bennée]

We don't include anything from qemu itself for the build.

Signed-off-by: Alex Bennée 
Reviewed-by: Thomas Huth 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: Richard Henderson 

---
v4
  - reword commit message to be more accurate
---
 tests/tcg/i386/test-i386.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/tcg/i386/test-i386.c b/tests/tcg/i386/test-i386.c
index 9599204895..cae6a7773a 100644
--- a/tests/tcg/i386/test-i386.c
+++ b/tests/tcg/i386/test-i386.c
@@ -17,7 +17,6 @@
  *  along with this program; if not, see .
  */
 #define _GNU_SOURCE
-#include "qemu/compiler.h"
 #include 
 #include 
 #include 
-- 
2.17.1

[Qemu-devel] [PATCH v7 25/54] tests/tcg: enable building for AArch64

[Alex Bennée]

We only have compilers for the (default) little endian variants.

Signed-off-by: Alex Bennée 
Reviewed-by: Richard Henderson 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v4
  - update MAINTAINERS
---
 MAINTAINERS| 1 +
 tests/tcg/aarch64/Makefile.include | 8 
 2 files changed, 9 insertions(+)
 create mode 100644 tests/tcg/aarch64/Makefile.include

diff --git a/MAINTAINERS b/MAINTAINERS
index 75a163f557..c4fc6fa357 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -136,6 +136,7 @@ L: qemu-...@nongnu.org
 S: Maintained
 F: target/arm/
 F: tests/tcg/arm/
+F: tests/tcg/aarch64/
 F: hw/arm/
 F: hw/cpu/a*mpcore.c
 F: include/hw/cpu/a*mpcore.h
diff --git a/tests/tcg/aarch64/Makefile.include 
b/tests/tcg/aarch64/Makefile.include
new file mode 100644
index 00..de32c91235
--- /dev/null
+++ b/tests/tcg/aarch64/Makefile.include
@@ -0,0 +1,8 @@
+# Makefile.include for AArch64 targets
+#
+# We don't have any bigendian build tools so we only use this for AArch64
+
+ifeq ($(TARGET_NAME),aarch64)
+DOCKER_IMAGE=debian-arm64-cross
+DOCKER_CROSS_COMPILER=aarch64-linux-gnu-gcc
+endif
-- 
2.17.1

[Qemu-devel] [PATCH v7 29/54] tests/docker/Makefile.include: fix mipsel-cross dependancy

[Alex Bennée]

This got broken in commit 4319db7 but generally only shows up when you
try and do massive parallel builds on fresh machines.

Signed-off-by: Alex Bennée 
---
 tests/docker/Makefile.include | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 8afb383478..4a2b028a3b 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -57,6 +57,7 @@ docker-image-debian-armel-cross: docker-image-debian9
 docker-image-debian-armhf-cross: docker-image-debian9
 docker-image-debian-arm64-cross: docker-image-debian9
 docker-image-debian-mips-cross: docker-image-debian9
+docker-image-debian-mipsel-cross: docker-image-debian9
 docker-image-debian-mips64el-cross: docker-image-debian9
 docker-image-debian-powerpc-cross: docker-image-debian8
 docker-image-debian-ppc64el-cross: docker-image-debian9
-- 
2.17.1

[Qemu-devel] [PATCH v7 33/54] tests/tcg: enable building for Alpha

[Alex Bennée]

We can't use our normal Debian based compilers as Alpha isn't an
officially supported architecture. However it is available as a port
and fortunately cross compilers for all these targets are included in
Debian Sid, the perpetual rolling/unstable/testing version of Debian.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v5
  - drop the unused debian-cross-build docker image
  - add note to debian-sid docker image
  - echo reference to notes on build failure
  - include EXTRA_RUNS for mmap tests
v6
  - use NOCACHE=1 for docker-image-debian-sid to force re-builds
  - this seems a little hacky
---
 tests/docker/Makefile.include |  7 
 .../dockerfiles/debian-alpha-cross.docker | 12 +++
 tests/docker/dockerfiles/debian-sid.docker| 32 +++
 tests/tcg/alpha/Makefile.include  |  2 ++
 tests/tcg/alpha/Makefile.target   |  6 
 5 files changed, 59 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-alpha-cross.docker
 create mode 100644 tests/docker/dockerfiles/debian-sid.docker
 create mode 100644 tests/tcg/alpha/Makefile.include
 create mode 100644 tests/tcg/alpha/Makefile.target

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 4a2b028a3b..8c24f8807e 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -64,6 +64,13 @@ docker-image-debian-ppc64el-cross: docker-image-debian9
 docker-image-debian-s390x-cross: docker-image-debian9
 docker-image-debian-win32-cross: docker-image-debian8-mxe
 docker-image-debian-win64-cross: docker-image-debian8-mxe
+
+# Debian SID images - we are tracking a rolling distro so we want to
+# force a re-build of the base image if we ever need to build one of
+# its children.
+docker-image-debian-sid: NOCACHE=1
+
+docker-image-debian-alpha-cross: docker-image-debian-sid
 docker-image-travis: NOUSER=1
 
 # Specialist build images, sometimes very limited tools
diff --git a/tests/docker/dockerfiles/debian-alpha-cross.docker 
b/tests/docker/dockerfiles/debian-alpha-cross.docker
new file mode 100644
index 00..29a25d0dfd
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-alpha-cross.docker
@@ -0,0 +1,12 @@
+#
+# Docker cross-compiler target
+#
+# This docker target builds on the debian sid base image which
+# contains cross compilers for Debian "ports" targets.
+#
+FROM qemu:debian-sid
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+apt-get install -y --no-install-recommends \
+gcc-alpha-linux-gnu \
+libc6.1-dev-alpha-cross || { echo "Failed to build - see 
debian-sid.docker notes"; exit 1; }
diff --git a/tests/docker/dockerfiles/debian-sid.docker 
b/tests/docker/dockerfiles/debian-sid.docker
new file mode 100644
index 00..9a3d168705
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-sid.docker
@@ -0,0 +1,32 @@
+#
+# Debian Sid Base
+#
+# A number of our guests exist as ports only. We can either use the
+# ports repo or get everything from Sid. However Sid is a rolling
+# distro which may be broken at any particular time. If you are
+# unlucky and try and build your images while gcc is in the process of
+# being uploaded this can fail. Your only recourse is to try again in
+# a few hours when the repos have re-synced. Once built however you
+# won't be affected by repo changes unless the docker recipies are
+# updated and trigger a re-build.
+#
+
+FROM debian:sid-slim
+
+# Duplicate deb line as deb-src
+RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> 
/etc/apt/sources.list
+
+# Install common build utilities
+RUN apt update
+RUN DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+apt install -y --no-install-recommends \
+bison \
+build-essential \
+ca-certificates \
+flex \
+git \
+pkg-config \
+psmisc \
+python \
+texinfo || { echo "Failed to build - see debian-sid.docker notes"; 
exit 1; }
diff --git a/tests/tcg/alpha/Makefile.include b/tests/tcg/alpha/Makefile.include
new file mode 100644
index 00..c7dc48eadb
--- /dev/null
+++ b/tests/tcg/alpha/Makefile.include
@@ -0,0 +1,2 @@
+DOCKER_IMAGE=debian-alpha-cross
+DOCKER_CROSS_COMPILER=alpha-linux-gnu-gcc
diff --git a/tests/tcg/alpha/Makefile.target b/tests/tcg/alpha/Makefile.target
new file mode 100644
index 00..bedf1b6444
--- /dev/null
+++ b/tests/tcg/alpha/Makefile.target
@@ -0,0 +1,6 @@
+# -*- Mode: makefile -*-
+#
+# Alpha specific tweaks
+
+# On Alpha Linux only supports 8k pages
+EXTRA_RUNS+=run-test-mmap-8192
-- 
2.17.1

[Qemu-devel] [PATCH v7 41/54] docker: move debian-powerpc-cross to sid based build

[Alex Bennée]

The original Jessie based cross builder hasn't worked for a while. The
state of the libraries is still perilous for cross-building QEMU but
we can use it for building TCG tests.

The debian-apt-fake.sh script can also be dropped as it is no longer
used.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
 tests/docker/Makefile.include |  4 +-
 tests/docker/dockerfiles/debian-apt-fake.sh   | 46 ---
 .../dockerfiles/debian-powerpc-cross.docker   | 39 +++-
 tests/docker/dockerfiles/debian8.docker   |  3 --
 4 files changed, 7 insertions(+), 85 deletions(-)
 delete mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index a680743ae8..87f449264f 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -47,8 +47,6 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
"BUILD","$*")
 
-docker-image-debian-powerpc-cross: 
EXTRA_FILES:=$(SRC_PATH)/tests/docker/dockerfiles/debian-apt-fake.sh
-
 # Enforce dependencies for composite images
 docker-image-debian: docker-image-debian9
 docker-image-debian8-mxe: docker-image-debian8
@@ -59,7 +57,6 @@ docker-image-debian-arm64-cross: docker-image-debian9
 docker-image-debian-mips-cross: docker-image-debian9
 docker-image-debian-mipsel-cross: docker-image-debian9
 docker-image-debian-mips64el-cross: docker-image-debian9
-docker-image-debian-powerpc-cross: docker-image-debian8
 docker-image-debian-ppc64el-cross: docker-image-debian9
 docker-image-debian-s390x-cross: docker-image-debian9
 docker-image-debian-win32-cross: docker-image-debian8-mxe
@@ -77,6 +74,7 @@ docker-image-debian-sh4-cross: docker-image-debian-sid
 docker-image-debian-sparc64-cross: docker-image-debian-sid
 docker-image-debian-mips64-cross: docker-image-debian-sid
 docker-image-debian-riscv64-cross: docker-image-debian-sid
+docker-image-debian-powerpc-cross: docker-image-debian-sid
 docker-image-travis: NOUSER=1
 
 # Specialist build images, sometimes very limited tools
diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh 
b/tests/docker/dockerfiles/debian-apt-fake.sh
deleted file mode 100755
index 2ec0fdf47a..00
--- a/tests/docker/dockerfiles/debian-apt-fake.sh
+++ /dev/null
@@ -1,46 +0,0 @@
-#! /bin/sh
-#
-# Generate fake debian package to resolve unimportant unmet dependencies held
-# by upstream multiarch broken packages.
-#
-# Copyright (c) 2017 Philippe Mathieu-Daudé 
-#
-# This work is licensed under the terms of the GNU GPL, version 2
-# or (at your option) any later version. See the COPYING file in
-# the top-level directory.
-
-test $1 = "install" && shift 1
-
-fake_install()
-{
-echo "Generating fake $2 $1 $3 ..."
-(cd /var/cache/apt/archives
-(cat << 'EOF'
-Section: misc
-Priority: optional
-Standards-Version: 3.9.2
-
-Package: NAME
-Version: VERSION
-Maintainer: qemu-devel@nongnu.org
-Architecture: any
-Multi-Arch: same
-Description: fake NAME
-EOF
-) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
-equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
-dpkg -i --force-overwrite $2_$3_$1.deb
-)
-}
-
-try_install()
-{
-name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
-arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
-vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
-apt-get install -q -yy $1 || fake_install $arch $name $vers
-}
-
-for package in $*; do
-try_install $package
-done
diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker 
b/tests/docker/dockerfiles/debian-powerpc-cross.docker
index a5dd46b4ac..5e62ca0df1 100644
--- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
+++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
@@ -1,40 +1,13 @@
 #
 # Docker powerpc cross-compiler target
 #
-# This docker target builds on the debian Jessie base image.
+# This docker target builds on the debian sid base image which
+# contains cross compilers for Debian "ports" targets. The original
+# Jessie based no longer builds.
 #
-FROM qemu:debian8
-MAINTAINER Philippe Mathieu-Daudé 
+FROM qemu:debian-sid
 
-# Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture powerpc
-RUN apt-get update
 RUN DEBIAN_FRONTEND=noninteractive eatmydata \
 apt-get install -y --no-install-recommends \
-crossbuild-essential-powerpc
-
-#  to fix "following packages have unmet dependencies" ...
-ADD debian-apt-fake.sh /usr/local/bin/apt-fake
-RUN apt-get install -y --no-install-recommends \
-equivs \
-pkg-config
-RUN apt-fake install \
-pkg-config:powerpc=0.28-1.1-fake && \
-ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
-ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
-# 
-
-# Specify the cross prefix for this image (see tests/docker/common.rc)
-ENV 

[Qemu-devel] [PATCH v7 16/54] tests/tcg/i386: Build fix for hello-i386

[Alex Bennée]

From: Fam Zheng 

We have -Werror=missing-prototype, add a dummy prototype to avoid that
warning.

Signed-off-by: Fam Zheng 
Reviewed-by: Thomas Huth 
Reviewed-by: Richard Henderson 
---
 tests/tcg/i386/hello-i386.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/tcg/i386/hello-i386.c b/tests/tcg/i386/hello-i386.c
index fa00380de2..cfeb24b2f5 100644
--- a/tests/tcg/i386/hello-i386.c
+++ b/tests/tcg/i386/hello-i386.c
@@ -20,6 +20,7 @@ static inline int write(int fd, const char * buf, int len)
   return status;
 }
 
+void _start(void);
 void _start(void)
 {
 write(1, "Hello World\n", 12);
-- 
2.17.1

[Qemu-devel] [PATCH v7 40/54] tests/tcg: enable building for RISCV64

[Alex Bennée]

As before, using Debian SID compilers.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
 tests/docker/Makefile.include|  1 +
 tests/docker/dockerfiles/debian-riscv64-cross.docker | 12 
 tests/tcg/riscv/Makefile.include | 10 ++
 3 files changed, 23 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-riscv64-cross.docker
 create mode 100644 tests/tcg/riscv/Makefile.include

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 8fd4041716..a680743ae8 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -76,6 +76,7 @@ docker-image-debian-m68k-cross: docker-image-debian-sid
 docker-image-debian-sh4-cross: docker-image-debian-sid
 docker-image-debian-sparc64-cross: docker-image-debian-sid
 docker-image-debian-mips64-cross: docker-image-debian-sid
+docker-image-debian-riscv64-cross: docker-image-debian-sid
 docker-image-travis: NOUSER=1
 
 # Specialist build images, sometimes very limited tools
diff --git a/tests/docker/dockerfiles/debian-riscv64-cross.docker 
b/tests/docker/dockerfiles/debian-riscv64-cross.docker
new file mode 100644
index 00..2b2e64cee6
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-riscv64-cross.docker
@@ -0,0 +1,12 @@
+#
+# Docker cross-compiler target
+#
+# This docker target builds on the debian sid base image which
+# contains cross compilers for Debian "ports" targets.
+#
+FROM qemu:debian-sid
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+apt-get install -y --no-install-recommends \
+gcc-riscv64-linux-gnu \
+libc6-dev-riscv64-cross
diff --git a/tests/tcg/riscv/Makefile.include b/tests/tcg/riscv/Makefile.include
new file mode 100644
index 00..d92ac6c89f
--- /dev/null
+++ b/tests/tcg/riscv/Makefile.include
@@ -0,0 +1,10 @@
+#
+# Makefile.include for all RISCV targets
+#
+# Debian only really cares about 64 bit going forward
+#
+
+ifeq ($(TARGET_NAME),riscv64)
+DOCKER_IMAGE=debian-riscv64-cross
+DOCKER_CROSS_COMPILER=riscv64-linux-gnu-gcc
+endif
-- 
2.17.1

[Qemu-devel] [PATCH v7 37/54] tests/tcg: enable building for sh4

[Alex Bennée]

As before, using Debian SID compilers. While the compiler can be
coerced into generating big-endian code it seems the linker can't deal
with it so we only enable the building for little endian SH4.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 

---
v5
   - add Makefile.target with EXTRA_RUNS for mmap tests
---
 tests/docker/Makefile.include|  1 +
 tests/docker/dockerfiles/debian-sh4-cross.docker | 12 
 tests/tcg/sh4/Makefile.include   |  4 
 tests/tcg/sh4/Makefile.target|  7 +++
 4 files changed, 24 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-sh4-cross.docker
 create mode 100644 tests/tcg/sh4/Makefile.include
 create mode 100644 tests/tcg/sh4/Makefile.target

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 59dd8e314d..a2dd307a8e 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -73,6 +73,7 @@ docker-image-debian-sid: NOCACHE=1
 docker-image-debian-alpha-cross: docker-image-debian-sid
 docker-image-debian-hppa-cross: docker-image-debian-sid
 docker-image-debian-m68k-cross: docker-image-debian-sid
+docker-image-debian-sh4-cross: docker-image-debian-sid
 docker-image-travis: NOUSER=1
 
 # Specialist build images, sometimes very limited tools
diff --git a/tests/docker/dockerfiles/debian-sh4-cross.docker 
b/tests/docker/dockerfiles/debian-sh4-cross.docker
new file mode 100644
index 00..88a2423094
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-sh4-cross.docker
@@ -0,0 +1,12 @@
+#
+# Docker cross-compiler target
+#
+# This docker target builds on the debian sid base image which
+# contains cross compilers for Debian "ports" targets.
+#
+FROM qemu:debian-sid
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+apt-get install -y --no-install-recommends \
+gcc-sh4-linux-gnu \
+libc6-dev-sh4-cross
diff --git a/tests/tcg/sh4/Makefile.include b/tests/tcg/sh4/Makefile.include
new file mode 100644
index 00..ad21594d9d
--- /dev/null
+++ b/tests/tcg/sh4/Makefile.include
@@ -0,0 +1,4 @@
+ifneq ($(TARGET_NAME), sh4eb)
+DOCKER_IMAGE=debian-sh4-cross
+DOCKER_CROSS_COMPILER=sh4-linux-gnu-gcc
+endif
diff --git a/tests/tcg/sh4/Makefile.target b/tests/tcg/sh4/Makefile.target
new file mode 100644
index 00..9d18d44612
--- /dev/null
+++ b/tests/tcg/sh4/Makefile.target
@@ -0,0 +1,7 @@
+# -*- Mode: makefile -*-
+#
+# SuperH specific tweaks
+#
+
+# On sh Linux supports 4k, 8k, 16k and 64k pages (but only 4k currently works)
+EXTRA_RUNS+=run-test-mmap-4096 # run-test-mmap-8192 run-test-mmap-16384 
run-test-mmap-65536
-- 
2.17.1

[Qemu-devel] [PATCH v7 54/54] .travis.yml: add check-tcg test

[Alex Bennée]

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 

---
v7
  - explicit build-tcg-prereqs
---
 .travis.yml | 8 
 1 file changed, 8 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index fabfe9ec34..ad352c5784 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -153,3 +153,11 @@ matrix:
 - TEST_CMD=""
   before_script:
 - ./configure ${CONFIG} --extra-cflags="-g3 -O0 -fsanitize=thread 
-fuse-ld=gold" || { cat config.log && exit 1; }
+- env:
+- CONFIG="--disable-system --disable-docs"
+- TEST_CMD="make check-tcg"
+  script:
+- make ${MAKEFLAGS} build-tcg-prereqs && ${TEST_CMD}
+  sudo: required
+  dist: trusty
+  compiler: gcc
-- 
2.17.1

[Qemu-devel] [PATCH v7 42/54] tests/tcg: enable building for PowerPC

[Alex Bennée]

Now we have restored debian-image-powerpc-cross using Debian SID
compilers we can build for 32 bit powerpc. Although PPC32 supports a
range of pages sizes currently only 4k works so the others are
commented out for now.

We can also merge the ppc64 support under the base architecture
directory to avoid too much proliferation of directories.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v5
  - new for v5
---
 tests/tcg/ppc/Makefile.include |  7 +++
 tests/tcg/ppc/Makefile.target  | 12 
 tests/tcg/ppc64le/Makefile.include |  2 --
 3 files changed, 19 insertions(+), 2 deletions(-)
 create mode 100644 tests/tcg/ppc/Makefile.include
 create mode 100644 tests/tcg/ppc/Makefile.target
 delete mode 100644 tests/tcg/ppc64le/Makefile.include

diff --git a/tests/tcg/ppc/Makefile.include b/tests/tcg/ppc/Makefile.include
new file mode 100644
index 00..b062c30dd3
--- /dev/null
+++ b/tests/tcg/ppc/Makefile.include
@@ -0,0 +1,7 @@
+ifeq ($(TARGET_NAME),ppc)
+DOCKER_IMAGE=debian-powerpc-cross
+DOCKER_CROSS_COMPILER=powerpc-linux-gnu-gcc
+else ifeq ($(TARGET_NAME),ppc64le)
+DOCKER_IMAGE=debian-ppc64el-cross
+DOCKER_CROSS_COMPILER=powerpc64le-linux-gnu-gcc
+endif
diff --git a/tests/tcg/ppc/Makefile.target b/tests/tcg/ppc/Makefile.target
new file mode 100644
index 00..f5e08c7376
--- /dev/null
+++ b/tests/tcg/ppc/Makefile.target
@@ -0,0 +1,12 @@
+# -*- Mode: makefile -*-
+#
+# PPC - included from tests/tcg/Makefile
+#
+
+ifneq (,$(findstring 64,$(TARGET_NAME)))
+# On PPC64 Linux can be configured with 4k (default) or 64k pages (currently 
broken)
+EXTRA_RUNS+=run-test-mmap-4096 #run-test-mmap-65536
+else
+# On PPC32 Linux supports 4K/16K/64K/256K (but currently only 4k works)
+EXTRA_RUNS+=run-test-mmap-4096 #run-test-mmap-16384 run-test-mmap-65536 
run-test-mmap-262144
+endif
diff --git a/tests/tcg/ppc64le/Makefile.include 
b/tests/tcg/ppc64le/Makefile.include
deleted file mode 100644
index d71cfc9aa7..00
--- a/tests/tcg/ppc64le/Makefile.include
+++ /dev/null
@@ -1,2 +0,0 @@
-DOCKER_IMAGE=debian-ppc64el-cross
-DOCKER_CROSS_COMPILER=powerpc64le-linux-gnu-gcc
-- 
2.17.1

[Qemu-devel] [PATCH v7 28/54] tests/tcg: enable building for MIPS

[Alex Bennée]

This doesn't add any additional tests but enables building the
multiarch tests for MIPS using docker cross compilers. We don't have a
cross compiler for mips64 big endian though.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
 tests/tcg/mips/Makefile.include | 17 +
 1 file changed, 17 insertions(+)
 create mode 100644 tests/tcg/mips/Makefile.include

diff --git a/tests/tcg/mips/Makefile.include b/tests/tcg/mips/Makefile.include
new file mode 100644
index 00..a9beceb623
--- /dev/null
+++ b/tests/tcg/mips/Makefile.include
@@ -0,0 +1,17 @@
+#
+# Makefile.include for all MIPs targets
+#
+# As Debian doesn't support mip64 in big endian mode the only way to
+# build BE is to pass a working cross compiler to ./configure
+#
+
+ifeq ($(TARGET_NAME),mips64el)
+DOCKER_IMAGE=debian-mips64el-cross
+DOCKER_CROSS_COMPILER=mips64el-linux-gnuabi64-gcc
+else ifeq ($(TARGET_NAME),mipsel)
+DOCKER_IMAGE=debian-mipsel-cross
+DOCKER_CROSS_COMPILER=mipsel-linux-gnu-gcc
+else ifeq ($(TARGET_NAME),mips)
+DOCKER_IMAGE=debian-mips-cross
+DOCKER_CROSS_COMPILER=mips-linux-gnu-gcc
+endif
-- 
2.17.1

[Qemu-devel] [PATCH v7 30/54] tests/tcg/mips: include common mips hello-mips

[Alex Bennée]

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
 tests/tcg/mips/Makefile.target | 19 +++
 1 file changed, 19 insertions(+)
 create mode 100644 tests/tcg/mips/Makefile.target

diff --git a/tests/tcg/mips/Makefile.target b/tests/tcg/mips/Makefile.target
new file mode 100644
index 00..71f0c2dd53
--- /dev/null
+++ b/tests/tcg/mips/Makefile.target
@@ -0,0 +1,19 @@
+# -*- Mode: makefile -*-
+#
+# MIPS - included from tests/tcg/Makefile.target
+#
+
+MIPS_SRC=$(SRC_PATH)/tests/tcg/mips
+
+# Set search path for all sources
+VPATH  += $(MIPS_SRC)
+
+MIPS_TESTS=hello-mips
+
+TESTS += $(MIPS_TESTS)
+
+hello-mips: CFLAGS+=-ffreestanding
+hello-mips: LDFLAGS+=-nostdlib
+
+# For MIPS32 and 64 we have a bunch of extra tests in sub-directories
+# however they are intended for system tests.
-- 
2.17.1

[Qemu-devel] [PATCH v7 19/54] tests/tcg/x86_64: add Makefile.target

[Alex Bennée]

The sources for x86_64 are shared in the i386 directory which will be
included thanks to TARGET_BASE_ARCH. However not all sources build so
we need to filter out the ones we can't build in the 64 bit world and
those that can't be built for 32 bit.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 

---
v4
  - update MAINTAINERS
v5
  - merge with disable i386 version of test-i386-ssse
---
 MAINTAINERS  |  1 +
 tests/tcg/i386/Makefile.target   |  4 ++--
 tests/tcg/x86_64/Makefile.target | 15 +++
 3 files changed, 18 insertions(+), 2 deletions(-)
 create mode 100644 tests/tcg/x86_64/Makefile.target

diff --git a/MAINTAINERS b/MAINTAINERS
index ea298cc910..57c828644f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -284,6 +284,7 @@ M: Eduardo Habkost 
 S: Maintained
 F: target/i386/
 F: tests/tcg/i386/
+F: tests/tcg/x86_64/
 F: hw/i386/
 F: disas/i386.c
 T: git git://github.com/ehabkost/qemu.git x86-next
diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target
index 64d241cfdf..cd173363ee 100644
--- a/tests/tcg/i386/Makefile.target
+++ b/tests/tcg/i386/Makefile.target
@@ -7,9 +7,9 @@ VPATH   += $(I386_SRC)
 
 I386_SRCS=$(notdir $(wildcard $(I386_SRC)/*.c))
 I386_TESTS=$(I386_SRCS:.c=)
-
+I386_ONLY_TESTS=$(filter-out test-i386-ssse3, $(I386_TESTS))
 # Update TESTS
-TESTS+=$(I386_TESTS)
+TESTS+=$(I386_ONLY_TESTS)
 
 ifneq ($(TARGET_NAME),x86_64)
 CFLAGS+=-m32
diff --git a/tests/tcg/x86_64/Makefile.target b/tests/tcg/x86_64/Makefile.target
new file mode 100644
index 00..74f170b9ed
--- /dev/null
+++ b/tests/tcg/x86_64/Makefile.target
@@ -0,0 +1,15 @@
+# -*- Mode: makefile -*-
+#
+# x86_64 tests - included from tests/tcg/Makefile.target
+#
+# Currently we only build test-x86_64 and test-i386-ssse3 from
+# $(SRC)/tests/tcg/i386/
+#
+
+X86_64_TESTS=$(filter-out $(I386_ONLY_TESTS), $(TESTS))
+X86_64_TESTS+=test-x86_64
+TESTS:=$(X86_64_TESTS)
+
+test-x86_64: LDFLAGS+=-lm -lc
+test-x86_64: test-i386.c test-i386.h test-i386-shift.h test-i386-muldiv.h
+   $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-- 
2.17.1

[Qemu-devel] [PATCH v7 22/54] tests/tcg: move ARM specific tests into subdir

[Alex Bennée]

These only need to be built for ARM guests.

Signed-off-by: Alex Bennée 
Reviewed-by: Thomas Huth 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: Richard Henderson 

---
v2
  - do VPATH manipulation in tests/tcg/arm/Makefile.target
  - merge with fix hello-arm test
v4
  - update MAINTAINERS
---
 MAINTAINERS   |  1 +
 tests/tcg/README  |  9 -
 tests/tcg/arm/Makefile.target | 13 +
 tests/tcg/arm/README  | 11 +++
 tests/tcg/{ => arm}/hello-arm.c   |  0
 tests/tcg/{ => arm}/test-arm-iwmmxt.s |  0
 6 files changed, 25 insertions(+), 9 deletions(-)
 create mode 100644 tests/tcg/arm/Makefile.target
 create mode 100644 tests/tcg/arm/README
 rename tests/tcg/{ => arm}/hello-arm.c (100%)
 rename tests/tcg/{ => arm}/test-arm-iwmmxt.s (100%)

diff --git a/MAINTAINERS b/MAINTAINERS
index 57c828644f..75a163f557 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -135,6 +135,7 @@ M: Peter Maydell 
 L: qemu-...@nongnu.org
 S: Maintained
 F: target/arm/
+F: tests/tcg/arm/
 F: hw/arm/
 F: hw/cpu/a*mpcore.c
 F: include/hw/cpu/a*mpcore.h
diff --git a/tests/tcg/README b/tests/tcg/README
index 469504c4cb..625f2326e6 100644
--- a/tests/tcg/README
+++ b/tests/tcg/README
@@ -5,15 +5,6 @@ or they are architecture specific.
 
 
 
-ARM
-===
-
-hello-arm
--
-
-test-arm-iwmmxt

-
 MIPS
 
 
diff --git a/tests/tcg/arm/Makefile.target b/tests/tcg/arm/Makefile.target
new file mode 100644
index 00..bc6962ecc6
--- /dev/null
+++ b/tests/tcg/arm/Makefile.target
@@ -0,0 +1,13 @@
+# -*- Mode: makefile -*-
+#
+# ARM - included from tests/tcg/Makefile.target
+#
+
+ARM_SRC=$(SRC_PATH)/tests/tcg/arm
+
+# Set search path for all sources
+VPATH  += $(ARM_SRC)
+
+
+hello-arm: CFLAGS+=-marm -ffreestanding
+hello-arm: LDFLAGS+=-nostdlib
diff --git a/tests/tcg/arm/README b/tests/tcg/arm/README
new file mode 100644
index 00..e6307116e2
--- /dev/null
+++ b/tests/tcg/arm/README
@@ -0,0 +1,11 @@
+These are ARM specific guest programs
+
+hello-arm
+-
+
+A very simple inline assembly, write syscall based hello world
+
+test-arm-iwmmxt
+---
+
+A simple test case for older iwmmxt extended ARMs
diff --git a/tests/tcg/hello-arm.c b/tests/tcg/arm/hello-arm.c
similarity index 100%
rename from tests/tcg/hello-arm.c
rename to tests/tcg/arm/hello-arm.c
diff --git a/tests/tcg/test-arm-iwmmxt.s b/tests/tcg/arm/test-arm-iwmmxt.s
similarity index 100%
rename from tests/tcg/test-arm-iwmmxt.s
rename to tests/tcg/arm/test-arm-iwmmxt.s
-- 
2.17.1

[Qemu-devel] [PATCH v7 49/54] docker: docker.py don't conflate checksums for extra_files

[Alex Bennée]

This just gets confusing especially as the helper function doesn't
even take into account any extra files (or the executable). Currently
the actual check just ignores them and also passes the result through
_dockerfile_preprocess so we fix that too.

Signed-off-by: Alex Bennée 
---
 tests/docker/docker.py | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index c83fd89985..2593a12d04 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -201,8 +201,10 @@ class Docker(object):
 
 tmp_df.write("\n")
 tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" %
- _text_checksum("\n".join([dockerfile] +
-extra_files_cksum)))
+ _text_checksum(_dockerfile_preprocess(dockerfile)))
+for f, c in extra_files_cksum:
+tmp_df.write("LABEL com.qemu.%s-checksum=%s" % (f, c))
+
 tmp_df.flush()
 
 self._do_check(["build", "-t", tag, "-f", tmp_df.name] + argv + \
@@ -317,7 +319,7 @@ class BuildCommand(SubCommand):
 _copy_binary_with_libs(args.include_executable, docker_dir)
 for filename in args.extra_files or []:
 _copy_with_mkdir(filename, docker_dir)
-cksum += [_file_checksum(filename)]
+cksum += [(filename, _file_checksum(filename))]
 
 argv += ["--build-arg=" + k.lower() + "=" + v
 for k, v in os.environ.iteritems()
-- 
2.17.1

[Qemu-devel] [PATCH v7 21/54] tests/tcg/i386/test-i386: fix printf format

[Alex Bennée]

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
 tests/tcg/i386/test-i386.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/tcg/i386/test-i386.c b/tests/tcg/i386/test-i386.c
index caef4da176..a29b41e764 100644
--- a/tests/tcg/i386/test-i386.c
+++ b/tests/tcg/i386/test-i386.c
@@ -2258,7 +2258,7 @@ SSE_OP(a ## sd);
 "pop %0\n"\
 : "=rm" (eflags)\
 : "x" (a.dq), "x" (b.dq));\
-printf("%-9s: a=%f b=%f cc=%04x\n",\
+printf("%-9s: a=%f b=%f cc=%04lx\n",\
#op, a1, b1,\
eflags & (CC_C | CC_P | CC_Z | CC_S | CC_O | CC_A));\
 }
-- 
2.17.1

[Qemu-devel] [PATCH v7 39/54] tests/tcg: enable building for mips64

[Alex Bennée]

As before, using Debian SID compilers.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
 tests/docker/Makefile.include   |  1 +
 tests/docker/dockerfiles/debian-mips64-cross.docker | 12 
 tests/tcg/mips/Makefile.include |  3 +++
 3 files changed, 16 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-mips64-cross.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index fb465e729e..8fd4041716 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -75,6 +75,7 @@ docker-image-debian-hppa-cross: docker-image-debian-sid
 docker-image-debian-m68k-cross: docker-image-debian-sid
 docker-image-debian-sh4-cross: docker-image-debian-sid
 docker-image-debian-sparc64-cross: docker-image-debian-sid
+docker-image-debian-mips64-cross: docker-image-debian-sid
 docker-image-travis: NOUSER=1
 
 # Specialist build images, sometimes very limited tools
diff --git a/tests/docker/dockerfiles/debian-mips64-cross.docker 
b/tests/docker/dockerfiles/debian-mips64-cross.docker
new file mode 100644
index 00..ed1ce0e919
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-mips64-cross.docker
@@ -0,0 +1,12 @@
+#
+# Docker cross-compiler target
+#
+# This docker target builds on the debian sid base image which
+# contains cross compilers for Debian "ports" targets.
+#
+FROM qemu:debian-sid
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata \
+apt-get install -y --no-install-recommends \
+gcc-mips64-linux-gnuabi64 \
+libc6-dev-mips64-cross
diff --git a/tests/tcg/mips/Makefile.include b/tests/tcg/mips/Makefile.include
index a9beceb623..4a14fc078d 100644
--- a/tests/tcg/mips/Makefile.include
+++ b/tests/tcg/mips/Makefile.include
@@ -8,6 +8,9 @@
 ifeq ($(TARGET_NAME),mips64el)
 DOCKER_IMAGE=debian-mips64el-cross
 DOCKER_CROSS_COMPILER=mips64el-linux-gnuabi64-gcc
+else ifeq ($(TARGET_NAME),mips64)
+DOCKER_IMAGE=debian-mips64-cross
+DOCKER_CROSS_COMPILER=mips64-linux-gnuabi64-gcc
 else ifeq ($(TARGET_NAME),mipsel)
 DOCKER_IMAGE=debian-mipsel-cross
 DOCKER_CROSS_COMPILER=mipsel-linux-gnu-gcc
-- 
2.17.1

[Qemu-devel] [PATCH v7 12/54] tests/tcg/multiarch: don't hard code paths/ports for linux-test

[Alex Bennée]

The fixed path and ports get in the way of running our tests and
builds in parallel. Instead of using TESTPATH we use mkdtemp() and
instead of a fixed port we allow the kernel to assign one and query it
afterwards.

Signed-off-by: Alex Bennée 
---
 tests/tcg/multiarch/linux-test.c | 37 
 1 file changed, 19 insertions(+), 18 deletions(-)

diff --git a/tests/tcg/multiarch/linux-test.c b/tests/tcg/multiarch/linux-test.c
index 6f2c531474..3f73b96420 100644
--- a/tests/tcg/multiarch/linux-test.c
+++ b/tests/tcg/multiarch/linux-test.c
@@ -41,8 +41,6 @@
 #include 
 #include 
 
-#define TESTPATH "/tmp/linux-test.tmp"
-#define TESTPORT 7654
 #define STACK_SIZE 16384
 
 static void error1(const char *filename, int line, const char *fmt, ...)
@@ -85,19 +83,15 @@ static void test_file(void)
 struct iovec vecs[2];
 DIR *dir;
 struct dirent *de;
+char template[] = "/tmp/linux-test-XX";
+char *tmpdir = mkdtemp(template);
 
-/* clean up, just in case */
-unlink(TESTPATH "/file1");
-unlink(TESTPATH "/file2");
-unlink(TESTPATH "/file3");
-rmdir(TESTPATH);
+chk_error(strlen(tmpdir));
 
 if (getcwd(cur_dir, sizeof(cur_dir)) == NULL)
 error("getcwd");
 
-chk_error(mkdir(TESTPATH, 0755));
-
-chk_error(chdir(TESTPATH));
+chk_error(chdir(tmpdir));
 
 /* open/read/write/close/readv/writev/lseek */
 
@@ -163,7 +157,7 @@ static void test_file(void)
 st.st_mtime != 1000)
 error("stat time");
 
-chk_error(stat(TESTPATH, ));
+chk_error(stat(tmpdir, ));
 if (!S_ISDIR(st.st_mode))
 error("stat mode");
 
@@ -185,7 +179,7 @@ static void test_file(void)
 error("stat mode");
 
 /* getdents */
-dir = opendir(TESTPATH);
+dir = opendir(tmpdir);
 if (!dir)
 error("opendir");
 len = 0;
@@ -207,7 +201,7 @@ static void test_file(void)
 chk_error(unlink("file3"));
 chk_error(unlink("file2"));
 chk_error(chdir(cur_dir));
-chk_error(rmdir(TESTPATH));
+chk_error(rmdir(tmpdir));
 }
 
 static void test_fork(void)
@@ -264,7 +258,7 @@ static int server_socket(void)
 chk_error(setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, , sizeof(val)));
 
 sockaddr.sin_family = AF_INET;
-sockaddr.sin_port = htons(TESTPORT);
+sockaddr.sin_port = htons(0); /* choose random ephemeral port) */
 sockaddr.sin_addr.s_addr = 0;
 chk_error(bind(fd, (struct sockaddr *), sizeof(sockaddr)));
 chk_error(listen(fd, 0));
@@ -272,7 +266,7 @@ static int server_socket(void)
 
 }
 
-static int client_socket(void)
+static int client_socket(uint16_t port)
 {
 int fd;
 struct sockaddr_in sockaddr;
@@ -280,7 +274,7 @@ static int client_socket(void)
 /* server socket */
 fd = chk_error(socket(PF_INET, SOCK_STREAM, 0));
 sockaddr.sin_family = AF_INET;
-sockaddr.sin_port = htons(TESTPORT);
+sockaddr.sin_port = htons(port);
 inet_aton("127.0.0.1", _addr);
 chk_error(connect(fd, (struct sockaddr *), sizeof(sockaddr)));
 return fd;
@@ -292,10 +286,17 @@ static void test_socket(void)
 {
 int server_fd, client_fd, fd, pid, ret, val;
 struct sockaddr_in sockaddr;
-socklen_t len;
+struct sockaddr_in server_addr;
+socklen_t len, socklen;
+uint16_t server_port;
 char buf[512];
 
 server_fd = server_socket();
+/* find out what port we got */
+socklen = sizeof(server_addr);
+ret = getsockname(server_fd, _addr, );
+chk_error(ret);
+server_port = ntohs(server_addr.sin_port);
 
 /* test a few socket options */
 len = sizeof(val);
@@ -305,7 +306,7 @@ static void test_socket(void)
 
 pid = chk_error(fork());
 if (pid == 0) {
-client_fd = client_socket();
+client_fd = client_socket(server_port);
 send(client_fd, socket_msg, sizeof(socket_msg), 0);
 close(client_fd);
 exit(0);
-- 
2.17.1

[Qemu-devel] [PATCH v7 20/54] tests/tcg/i386/test-i386: use modern vector_size attributes

[Alex Bennée]

The compiler complains about the old __mode__ style attributes.

Signed-off-by: Alex Bennée 
Reviewed-by: Richard Henderson 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
 tests/tcg/i386/test-i386.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/tcg/i386/test-i386.c b/tests/tcg/i386/test-i386.c
index cae6a7773a..caef4da176 100644
--- a/tests/tcg/i386/test-i386.c
+++ b/tests/tcg/i386/test-i386.c
@@ -2106,8 +2106,8 @@ static void test_enter(void)
 
 #ifdef TEST_SSE
 
-typedef int __m64 __attribute__ ((__mode__ (__V2SI__)));
-typedef float __m128 __attribute__ ((__mode__(__V4SF__)));
+typedef int __m64 __attribute__ ((vector_size(8)));
+typedef float __m128 __attribute__ ((vector_size(16)));
 
 typedef union {
 double d[2];
-- 
2.17.1

[Qemu-devel] [PATCH v7 01/54] configure: add support for --cross-cc-FOO

[Alex Bennée]

This allows us to specify cross compilers for our guests. This is
useful for building test images/programs. Currently we re-run the
compile test for each target. I couldn't think of a way to cache the
value for a given arch without getting messier configure code.

The cross compiler for the guest is visible to each target as
CROSS_CC_GUEST in config-target.mak.

Signed-off-by: Alex Bennée 
Reviewed-by: Richard Henderson 
---
v3
  - --cross-cc-*[!a-zA-Z0-9_-]*=*) error_exit...
  - --cross-cc-*) cc_arch=${opt#--cross-cc-}; cc_arch=${cc_arch%%=*}
  - add remaining target_compiler definitions
v4
  - also set cross_cc_$cpu=$host_cc
  - try a non-static build if static fails
  - use write_c_skeleton
  - add doc string in help
---
 configure | 92 +++
 1 file changed, 92 insertions(+)

diff --git a/configure b/configure
index a8c4094c87..089de03918 100755
--- a/configure
+++ b/configure
@@ -458,6 +458,13 @@ vxhs=""
 libxml2=""
 docker="no"
 
+# cross compilers defaults, can be overridden with --cross-cc-ARCH
+cross_cc_aarch64="aarch64-linux-gnu-gcc"
+cross_cc_arm="arm-linux-gnueabihf-gcc"
+cross_cc_powerpc="powerpc-linux-gnu-gcc"
+
+enabled_cross_compilers=""
+
 supported_cpu="no"
 supported_os="no"
 bogus_os="no"
@@ -488,6 +495,11 @@ for opt do
   ;;
   --disable-debug-info) debug_info="no"
   ;;
+  --cross-cc-*[!a-zA-Z0-9_-]*=*) error_exit "Passed bad --cross-cc-FOO option"
+  ;;
+  --cross-cc-*) cc_arch=${opt#--cross-cc-}; cc_arch=${cc_arch%%=*}
+eval "cross_cc_${cc_arch}=\$optarg"
+  ;;
   esac
 done
 # OS specific
@@ -676,30 +688,37 @@ case "$cpu" in
   ppc|ppc64|s390|s390x|sparc64|x32)
 cpu="$cpu"
 supported_cpu="yes"
+eval "cross_cc_${cpu}=\$host_cc"
   ;;
   i386|i486|i586|i686|i86pc|BePC)
 cpu="i386"
 supported_cpu="yes"
+cross_cc_i386=$host_cc
   ;;
   x86_64|amd64)
 cpu="x86_64"
 supported_cpu="yes"
+cross_cc_x86_64=$host_cc
   ;;
   armv*b|armv*l|arm)
 cpu="arm"
 supported_cpu="yes"
+cross_cc_arm=$host_cc
   ;;
   aarch64)
 cpu="aarch64"
 supported_cpu="yes"
+cross_cc_aarch64=$host_cc
   ;;
   mips*)
 cpu="mips"
 supported_cpu="yes"
+cross_cc_mips=$host_cc
   ;;
   sparc|sun4[cdmuv])
 cpu="sparc"
 supported_cpu="yes"
+cross_cc_sparc=$host_cc
   ;;
   *)
 # This will result in either an error or falling back to TCI later
@@ -917,6 +936,8 @@ for opt do
   ;;
   --disable-debug-info)
   ;;
+  --cross-cc-*)
+  ;;
   --enable-modules)
   modules="yes"
   ;;
@@ -1501,6 +1522,7 @@ Advanced options (experts only):
   --extra-cflags=CFLAGSappend extra C compiler flags QEMU_CFLAGS
   --extra-cxxflags=CXXFLAGS append extra C++ compiler flags QEMU_CXXFLAGS
   --extra-ldflags=LDFLAGS  append extra linker flags LDFLAGS
+  --cross-cc-ARCH=CC   use compiler when building ARCH guest test cases
   --make=MAKE  use specified make [$make]
   --install=INSTALLuse specified install [$install]
   --python=PYTHON  use specified python [$python]
@@ -6820,6 +6842,9 @@ case "$target" in
 ;;
 esac
 
+target_compiler=""
+target_compiler_static=""
+
 mkdir -p $target_dir
 echo "# Automatically generated by configure - do not modify" > 
$config_target_mak
 
@@ -6835,19 +6860,23 @@ TARGET_ABI_DIR=""
 case "$target_name" in
   i386)
 gdb_xml_files="i386-32bit.xml i386-32bit-core.xml i386-32bit-sse.xml"
+target_compiler=$cross_cc_i386
   ;;
   x86_64)
 TARGET_BASE_ARCH=i386
 gdb_xml_files="i386-64bit.xml i386-64bit-core.xml i386-64bit-sse.xml"
+target_compiler=$cross_cc_x86_64
   ;;
   alpha)
 mttcg="yes"
+target_compiler=$cross_cc_alpha
   ;;
   arm|armeb)
 TARGET_ARCH=arm
 bflt="yes"
 mttcg="yes"
 gdb_xml_files="arm-core.xml arm-vfp.xml arm-vfp3.xml arm-neon.xml"
+target_compiler=$cross_cc_arm
   ;;
   aarch64|aarch64_be)
 TARGET_ARCH=aarch64
@@ -6855,59 +6884,74 @@ case "$target_name" in
 bflt="yes"
 mttcg="yes"
 gdb_xml_files="aarch64-core.xml aarch64-fpu.xml arm-core.xml arm-vfp.xml 
arm-vfp3.xml arm-neon.xml"
+target_compiler=$cross_cc_aarch64
   ;;
   cris)
+target_compiler=$cross_cc_cris
   ;;
   hppa)
 mttcg="yes"
+target_compiler=$cross_cc_hppa
   ;;
   lm32)
+target_compiler=$cross_cc_lm32
   ;;
   m68k)
 bflt="yes"
 gdb_xml_files="cf-core.xml cf-fp.xml m68k-fp.xml"
+target_compiler=$cross_cc_m68k
   ;;
   microblaze|microblazeel)
 TARGET_ARCH=microblaze
 bflt="yes"
 echo "TARGET_ABI32=y" >> $config_target_mak
+target_compiler=$cross_cc_microblaze
   ;;
   mips|mipsel)
 TARGET_ARCH=mips
+target_compiler=$cross_cc_mips
 echo "TARGET_ABI_MIPSO32=y" >> $config_target_mak
   ;;
   mipsn32|mipsn32el)
 TARGET_ARCH=mips64
 TARGET_BASE_ARCH=mips
+target_compiler=$cross_cc_mipsn32
 echo "TARGET_ABI_MIPSN32=y" >> $config_target_mak
 echo "TARGET_ABI32=y" >> $config_target_mak
   ;;
   

[Qemu-devel] [PATCH v7 10/54] tests/tcg/multiarch: Build fix for linux-test

[Alex Bennée]

From: Fam Zheng 

To keep the compiler happy, and to fit in our buildsys flags:

- Make local functions "static"
- #ifdef out unused functions
- drop cutils/osdep dependencies

Signed-off-by: Fam Zheng 
[AJB: drop cutils/osdep dependencies]
Signed-off-by: Alex Bennée 
Reviewed-by: Thomas Huth 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: Richard Henderson 
---
 tests/tcg/multiarch/linux-test.c | 68 ++--
 1 file changed, 21 insertions(+), 47 deletions(-)

diff --git a/tests/tcg/multiarch/linux-test.c b/tests/tcg/multiarch/linux-test.c
index 5070d31446..4457bd04ba 100644
--- a/tests/tcg/multiarch/linux-test.c
+++ b/tests/tcg/multiarch/linux-test.c
@@ -16,7 +16,6 @@
  *  You should have received a copy of the GNU General Public License
  *  along with this program; if not, see .
  */
-#define _GNU_SOURCE
 #include 
 #include 
 #include 
@@ -31,6 +30,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -39,13 +39,12 @@
 #include 
 #include 
 #include 
-#include "qemu/cutils.h"
 
 #define TESTPATH "/tmp/linux-test.tmp"
 #define TESTPORT 7654
 #define STACK_SIZE 16384
 
-void error1(const char *filename, int line, const char *fmt, ...)
+static void error1(const char *filename, int line, const char *fmt, ...)
 {
 va_list ap;
 va_start(ap, fmt);
@@ -56,7 +55,7 @@ void error1(const char *filename, int line, const char *fmt, 
...)
 exit(1);
 }
 
-int __chk_error(const char *filename, int line, int ret)
+static int __chk_error(const char *filename, int line, int ret)
 {
 if (ret < 0) {
 error1(filename, line, "%m (ret=%d, errno=%d)",
@@ -73,7 +72,7 @@ int __chk_error(const char *filename, int line, int ret)
 
 #define FILE_BUF_SIZE 300
 
-void test_file(void)
+static void test_file(void)
 {
 int fd, i, len, ret;
 uint8_t buf[FILE_BUF_SIZE];
@@ -210,7 +209,7 @@ void test_file(void)
 chk_error(rmdir(TESTPATH));
 }
 
-void test_fork(void)
+static void test_fork(void)
 {
 int pid, status;
 
@@ -224,7 +223,7 @@ void test_fork(void)
 error("waitpid status=0x%x", status);
 }
 
-void test_time(void)
+static void test_time(void)
 {
 struct timeval tv, tv2;
 struct timespec ts, rem;
@@ -251,34 +250,7 @@ void test_time(void)
 error("getrusage");
 }
 
-void pstrcpy(char *buf, int buf_size, const char *str)
-{
-int c;
-char *q = buf;
-
-if (buf_size <= 0)
-return;
-
-for(;;) {
-c = *str++;
-if (c == 0 || q >= buf + buf_size - 1)
-break;
-*q++ = c;
-}
-*q = '\0';
-}
-
-/* strcat and truncate. */
-char *pstrcat(char *buf, int buf_size, const char *s)
-{
-int len;
-len = strlen(buf);
-if (len < buf_size)
-pstrcpy(buf + len, buf_size - len, s);
-return buf;
-}
-
-int server_socket(void)
+static int server_socket(void)
 {
 int val, fd;
 struct sockaddr_in sockaddr;
@@ -298,7 +270,7 @@ int server_socket(void)
 
 }
 
-int client_socket(void)
+static int client_socket(void)
 {
 int fd;
 struct sockaddr_in sockaddr;
@@ -312,9 +284,9 @@ int client_socket(void)
 return fd;
 }
 
-const char socket_msg[] = "hello socket\n";
+static const char socket_msg[] = "hello socket\n";
 
-void test_socket(void)
+static void test_socket(void)
 {
 int server_fd, client_fd, fd, pid, ret, val;
 struct sockaddr_in sockaddr;
@@ -348,9 +320,10 @@ void test_socket(void)
 chk_error(close(server_fd));
 }
 
+#if 0
 #define WCOUNT_MAX 512
 
-void test_pipe(void)
+static void test_pipe(void)
 {
 fd_set rfds, wfds;
 int fds[2], fd_max, ret;
@@ -391,10 +364,10 @@ void test_pipe(void)
 chk_error(close(fds[1]));
 }
 
-int thread1_res;
-int thread2_res;
+static int thread1_res;
+static int thread2_res;
 
-int thread1_func(void *arg)
+static int thread1_func(void *arg)
 {
 int i;
 for(i=0;i<5;i++) {
@@ -404,7 +377,7 @@ int thread1_func(void *arg)
 return 0;
 }
 
-int thread2_func(void *arg)
+static int thread2_func(void *arg)
 {
 int i;
 for(i=0;i<6;i++) {
@@ -435,27 +408,28 @@ void test_clone(void)
 thread2_res != 6)
 error("clone");
 }
+#endif
 
 /***/
 
 volatile int alarm_count;
 jmp_buf jmp_env;
 
-void sig_alarm(int sig)
+static void sig_alarm(int sig)
 {
 if (sig != SIGALRM)
 error("signal");
 alarm_count++;
 }
 
-void sig_segv(int sig, siginfo_t *info, void *puc)
+static void sig_segv(int sig, siginfo_t *info, void *puc)
 {
 if (sig != SIGSEGV)
 error("signal");
 longjmp(jmp_env, 1);
 }
 
-void test_signal(void)
+static void test_signal(void)
 {
 struct sigaction act;
 struct itimerval it, oit;
@@ -510,7 +484,7 @@ void test_signal(void)
 
 #define SHM_SIZE 32768
 
-void test_shm(void)
+static void test_shm(void)
 {
 void *ptr;
 int shmid;
-- 
2.17.1

[Qemu-devel] [PATCH v7 48/54] tests: add top-level make dependency for docker builds

[Alex Bennée]

One problem with satisfying your docker dependencies in a sub-make it
you might end up trying to satisfy the dependency multiple times. This
is especially a problem with debian-sid based cross compilers and CI
setups. We solve this by doing a docker build pass at the top level
before any sub-makes are called.

We still need to satisfy dependencies in the Makefile.target call so
people can run tests from individual target directories. We introduce
a new Makefile.probe which gets called for each PROBE_TARGET and
allows us to build up the list. It does require multiply including
config-target.mak which shouldn't cause any issues as it shouldn't
define anything that clashes with config-host.mak. However we undefine
a few key variables each time around.

Signed-off-by: Alex Bennée 

---
v7
  - remove $(info) spam
  - just add the explicit docker dependency to each build target
  - include build-tcg-prereqs for CI tasks
---
 tests/Makefile.include   | 11 +++
 tests/tcg/Makefile.probe | 31 +++
 2 files changed, 42 insertions(+)
 create mode 100644 tests/tcg/Makefile.probe

diff --git a/tests/Makefile.include b/tests/Makefile.include
index c8faa01a33..2dd0700a88 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -938,6 +938,14 @@ BUILD_TCG_TARGET_RULES=$(patsubst %,build-tcg-tests-%, 
$(LINUX_USER_TARGETS))
 CLEAN_TCG_TARGET_RULES=$(patsubst %,clean-tcg-tests-%, $(LINUX_USER_TARGETS))
 RUN_TCG_TARGET_RULES=$(patsubst %,run-tcg-tests-%, $(LINUX_USER_TARGETS))
 
+# Probe for the Docker Builds needed for each build
+DOCKER_BUILD_TARGETS:=
+$(foreach PROBE_TARGET,$(TARGET_LIST), \
+   $(eval -include $(SRC_PATH)/tests/tcg/Makefile.probe)   \
+   $(if $(DOCKER_PREREQ),  \
+   $(eval DOCKER_BUILD_TARGETS+=$(DOCKER_PREREQ))  \
+   $(eval build-tcg-tests-$(PROBE_TARGET): $(DOCKER_PREREQ
+
 build-tcg-tests-%:
$(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" 
TARGET_DIR="$*/" guest-tests,)
 
@@ -947,6 +955,9 @@ run-tcg-tests-%: build-tcg-tests-%
 clean-tcg-tests-%:
$(call quiet-command,$(MAKE) $(SUBDIR_MAKEFLAGS) -C $* V="$(V)" 
TARGET_DIR="$*/" clean-guest-tests,)
 
+.PHONY: build-tcg-prereqs
+build-tcg-prereqs: $(DOCKER_BUILD_TARGETS)
+
 .PHONY: build-tcg
 build-tcg: $(BUILD_TCG_TARGET_RULES)
 
diff --git a/tests/tcg/Makefile.probe b/tests/tcg/Makefile.probe
new file mode 100644
index 00..15c0412657
--- /dev/null
+++ b/tests/tcg/Makefile.probe
@@ -0,0 +1,31 @@
+# -*- Mode: makefile -*-
+#
+# TCG Compiler Probe
+#
+# This Makefile fragement is included multiple times in the main make
+# script to probe for available compilers. This is used to build up a
+# selection of required docker targets before we invoke a sub-make for
+# each target.
+
+# First we need the target makefile which tells us the target architecture
+-include $(BUILD_DIR)/$(PROBE_TARGET)/config-target.mak
+
+# Then we load up the target architecture makefiles which tell us
+# about the compilers
+CROSS_CC_GUEST:=
+DOCKER_IMAGE:=
+DOCKER_PREREQ:=
+
+-include $(SRC_PATH)/tests/tcg/$(TARGET_BASE_ARCH)/Makefile.include
+-include $(SRC_PATH)/tests/tcg/$(TARGET_NAME)/Makefile.include
+
+ifndef CROSS_CC_GUEST
+ifneq ($(DOCKER_IMAGE),)
+DOCKER_PREREQ:=docker-image-$(DOCKER_IMAGE)
+endif
+endif
+
+# Clean-up
+# undefine TARGET_NAME
+# undefine TARGET_BASE_ARCH
+# undefine TARGET_ABI_DIR
-- 
2.17.1

[Qemu-devel] [PATCH v7 02/54] configure: move i386_cc to cross_cc_i386

[Alex Bennée]

Also dont assume x86_64 compiler can build i386 binaries.

Signed-off-by: Alex Bennée 
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: Richard Henderson 
---
v2
  - drop using system x86_64 compiler for i386 compiles
v3
  - fix cross_cc_i386 when on i386
v4
  - fix merge conflicts
  - sort cross_cc defines
---
 configure | 24 +---
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/configure b/configure
index 089de03918..6e33dbbe10 100755
--- a/configure
+++ b/configure
@@ -289,7 +289,6 @@ libs_softmmu=""
 libs_tools=""
 audio_pt_int=""
 audio_win_int=""
-cc_i386=i386-pc-linux-gnu-gcc
 libs_qga=""
 debug_info="yes"
 stack_protector=""
@@ -461,6 +460,8 @@ docker="no"
 # cross compilers defaults, can be overridden with --cross-cc-ARCH
 cross_cc_aarch64="aarch64-linux-gnu-gcc"
 cross_cc_arm="arm-linux-gnueabihf-gcc"
+cross_cc_i386="i386-pc-linux-gnu-gcc"
+cross_cc_cflags_i386=""
 cross_cc_powerpc="powerpc-linux-gnu-gcc"
 
 enabled_cross_compilers=""
@@ -1447,7 +1448,8 @@ case "$cpu" in
 i386)
CPU_CFLAGS="-m32"
LDFLAGS="-m32 $LDFLAGS"
-   cc_i386='$(CC) -m32'
+   cross_cc_i386=$cc
+   cross_cc_cflags_i386=$CPU_CFLAGS
;;
 x86_64)
# ??? Only extremely old AMD cpus do not have cmpxchg16b.
@@ -1455,12 +1457,14 @@ case "$cpu" in
# runtime and generate the fallback to serial emulation.
CPU_CFLAGS="-m64 -mcx16"
LDFLAGS="-m64 $LDFLAGS"
-   cc_i386='$(CC) -m32'
+   cross_cc_x86_64=$cc
+   cross_cc_cflags_x86_64=$CPU_CFLAGS
;;
 x32)
CPU_CFLAGS="-mx32"
LDFLAGS="-mx32 $LDFLAGS"
-   cc_i386='$(CC) -m32'
+   cross_cc_i386=$cc
+   cross_cc_cflags_i386="-m32"
;;
 # No special flags required for other host CPUs
 esac
@@ -6723,7 +6727,6 @@ echo "CC=$cc" >> $config_host_mak
 if $iasl -h > /dev/null 2>&1; then
   echo "IASL=$iasl" >> $config_host_mak
 fi
-echo "CC_I386=$cc_i386" >> $config_host_mak
 echo "HOST_CC=$host_cc" >> $config_host_mak
 echo "CXX=$cxx" >> $config_host_mak
 echo "OBJCC=$objcc" >> $config_host_mak
@@ -6844,6 +6847,7 @@ esac
 
 target_compiler=""
 target_compiler_static=""
+target_compiler_cflags=""
 
 mkdir -p $target_dir
 echo "# Automatically generated by configure - do not modify" > 
$config_target_mak
@@ -6861,6 +6865,7 @@ case "$target_name" in
   i386)
 gdb_xml_files="i386-32bit.xml i386-32bit-core.xml i386-32bit-sse.xml"
 target_compiler=$cross_cc_i386
+target_compiler_cflags=$cross_cc_ccflags_i386
   ;;
   x86_64)
 TARGET_BASE_ARCH=i386
@@ -7033,9 +7038,9 @@ if has $target_compiler; then
 
 write_c_skeleton
 
-if ! do_compiler $target_compiler -o $TMPE $TMPC -static ; then
+if ! do_compiler $target_compiler $target_compiler_cflags -o $TMPE $TMPC 
-static ; then
 # For host systems we might get away with building without -static
-if ! do_compiler $target_compiler -o $TMPE $TMPC ; then
+if ! do_compiler $target_compiler $target_compiler_cflags -o $TMPE 
$TMPC ; then
 target_compiler=""
 else
 enabled_cross_compilers="${enabled_cross_compilers} 
${target_compiler}"
@@ -7128,8 +7133,13 @@ if test -n "$target_compiler"; then
   if test -n "$target_compiler_static"; then
   echo "CROSS_CC_GUEST_STATIC=$target_compiler_static" >> 
$config_target_mak
   fi
+
+  if test -n "$target_compiler_cflags"; then
+  echo "CROSS_CC_GUEST_CFLAGS=$target_compiler_cflags" >> 
$config_target_mak
+  fi
 fi
 
+
 # generate QEMU_CFLAGS/LDFLAGS for targets
 
 cflags=""
-- 
2.17.1

[Qemu-devel] [PATCH v7 11/54] tests/tcg/multiarch: enable additional linux-test tests

[Alex Bennée]

Un-comment the remaining tests.

I removed the itimer value tests because I'm fairly sure a re-arming
timer will always have a different value in it when you grab it.

I've also fixed up the clone thread flags as QEMU will only allow a
clone to use flags which match glibc. However the test is still racey
so it remains disabled by default - it can be run by passing any
additional parameters on the command line.

Signed-off-by: Alex Bennée 
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: Richard Henderson 
---
v3
  - make clone flags match CLONE_THREAD_FLAGS
v4
  - disable test_clone for default case
---
 tests/tcg/multiarch/linux-test.c | 51 +---
 1 file changed, 33 insertions(+), 18 deletions(-)

diff --git a/tests/tcg/multiarch/linux-test.c b/tests/tcg/multiarch/linux-test.c
index 4457bd04ba..6f2c531474 100644
--- a/tests/tcg/multiarch/linux-test.c
+++ b/tests/tcg/multiarch/linux-test.c
@@ -16,6 +16,7 @@
  *  You should have received a copy of the GNU General Public License
  *  along with this program; if not, see .
  */
+#define _GNU_SOURCE
 #include 
 #include 
 #include 
@@ -58,8 +59,8 @@ static void error1(const char *filename, int line, const char 
*fmt, ...)
 static int __chk_error(const char *filename, int line, int ret)
 {
 if (ret < 0) {
-error1(filename, line, "%m (ret=%d, errno=%d)",
-   ret, errno);
+error1(filename, line, "%m (ret=%d, errno=%d/%s)",
+   ret, errno, strerror(errno));
 }
 return ret;
 }
@@ -216,6 +217,7 @@ static void test_fork(void)
 pid = chk_error(fork());
 if (pid == 0) {
 /* child */
+sleep(2);
 exit(2);
 }
 chk_error(waitpid(pid, , 0));
@@ -320,7 +322,6 @@ static void test_socket(void)
 chk_error(close(server_fd));
 }
 
-#if 0
 #define WCOUNT_MAX 512
 
 static void test_pipe(void)
@@ -355,7 +356,7 @@ static void test_pipe(void)
 }
 if (FD_ISSET(fds[1], )) {
 ch = 'a';
-chk_error(write(fds[0], , 1));
+chk_error(write(fds[1], , 1));
 wcount++;
 }
 }
@@ -387,28 +388,41 @@ static int thread2_func(void *arg)
 return 0;
 }
 
-void test_clone(void)
+static void wait_for_child(pid_t pid)
+{
+int status;
+chk_error(waitpid(pid, , 0));
+}
+
+/* For test_clone we must match the clone flags used by glibc, see
+ * CLONE_THREAD_FLAGS in the QEMU source code.
+ */
+static void test_clone(void)
 {
 uint8_t *stack1, *stack2;
-int pid1, pid2, status1, status2;
+pid_t pid1, pid2;
 
 stack1 = malloc(STACK_SIZE);
 pid1 = chk_error(clone(thread1_func, stack1 + STACK_SIZE,
-   CLONE_VM | CLONE_FS | CLONE_FILES | SIGCHLD, 
"hello1"));
+   CLONE_VM | CLONE_FS | CLONE_FILES |
+   CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM,
+"hello1"));
 
 stack2 = malloc(STACK_SIZE);
 pid2 = chk_error(clone(thread2_func, stack2 + STACK_SIZE,
-   CLONE_VM | CLONE_FS | CLONE_FILES | SIGCHLD, 
"hello2"));
+   CLONE_VM | CLONE_FS | CLONE_FILES |
+   CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM,
+   "hello2"));
 
-while (waitpid(pid1, , 0) != pid1);
+wait_for_child(pid1);
 free(stack1);
-while (waitpid(pid2, , 0) != pid2);
+wait_for_child(pid2);
 free(stack2);
+
 if (thread1_res != 5 ||
 thread2_res != 6)
 error("clone");
 }
-#endif
 
 /***/
 
@@ -449,12 +463,10 @@ static void test_signal(void)
 it.it_value.tv_usec = 10 * 1000;
 chk_error(setitimer(ITIMER_REAL, , NULL));
 chk_error(getitimer(ITIMER_REAL, ));
-if (oit.it_value.tv_sec != it.it_value.tv_sec ||
-oit.it_value.tv_usec != it.it_value.tv_usec)
-error("itimer");
 
 while (alarm_count < 5) {
 usleep(10 * 1000);
+getitimer(ITIMER_REAL, );
 }
 
 it.it_interval.tv_sec = 0;
@@ -463,9 +475,6 @@ static void test_signal(void)
 it.it_value.tv_usec = 0;
 memset(, 0xff, sizeof(oit));
 chk_error(setitimer(ITIMER_REAL, , ));
-if (oit.it_value.tv_sec != 0 ||
-oit.it_value.tv_usec != 10 * 1000)
-error("setitimer");
 
 /* SIGSEGV test */
 act.sa_sigaction = sig_segv;
@@ -503,10 +512,16 @@ static void test_shm(void)
 int main(int argc, char **argv)
 {
 test_file();
+test_pipe();
 test_fork();
 test_time();
 test_socket();
-//test_clone();
+
+if (argc > 1) {
+printf("test_clone still considered buggy\n");
+test_clone();
+}
+
 test_signal();
 test_shm();
 return 0;
-- 
2.17.1

[Qemu-devel] [PATCH v7 31/54] tests/tcg: enable building for s390x

[Alex Bennée]

This doesn't add any additional tests but enables building the
multiarch tests for s390x.

Signed-off-by: Alex Bennée 
Acked-by: Cornelia Huck 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: David Hildenbrand 
---
 tests/tcg/s390x/Makefile.include | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 tests/tcg/s390x/Makefile.include

diff --git a/tests/tcg/s390x/Makefile.include b/tests/tcg/s390x/Makefile.include
new file mode 100644
index 00..1f58115d96
--- /dev/null
+++ b/tests/tcg/s390x/Makefile.include
@@ -0,0 +1,2 @@
+DOCKER_IMAGE=debian-s390x-cross
+DOCKER_CROSS_COMPILER=s390x-linux-gnu-gcc
-- 
2.17.1

[Qemu-devel] [PULL v2 16/19] cputlb: remove tb_lock from tlb_flush functions

[Richard Henderson]

From: "Emilio G. Cota" 

The acquisition of tb_lock was added when the async tlb_flush
was introduced in e3b9ca810 ("cputlb: introduce tlb_flush_* async work.")

tb_lock was there to allow us to do memset() on the tb_jmp_cache's.
However, since f3ced3c5928 ("tcg: consistently access cpu->tb_jmp_cache
atomically") all accesses to tb_jmp_cache are atomic, so tb_lock
is not needed here. Get rid of it.

Reviewed-by: Alex Bennée 
Reviewed-by: Richard Henderson 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 accel/tcg/cputlb.c | 8 
 1 file changed, 8 deletions(-)

diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 0a721bb9c4..719cca2268 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -125,8 +125,6 @@ static void tlb_flush_nocheck(CPUState *cpu)
 atomic_set(>tlb_flush_count, env->tlb_flush_count + 1);
 tlb_debug("(count: %zu)\n", tlb_flush_count());
 
-tb_lock();
-
 memset(env->tlb_table, -1, sizeof(env->tlb_table));
 memset(env->tlb_v_table, -1, sizeof(env->tlb_v_table));
 cpu_tb_jmp_cache_clear(cpu);
@@ -135,8 +133,6 @@ static void tlb_flush_nocheck(CPUState *cpu)
 env->tlb_flush_addr = -1;
 env->tlb_flush_mask = 0;
 
-tb_unlock();
-
 atomic_mb_set(>pending_tlb_flush, 0);
 }
 
@@ -180,8 +176,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, 
run_on_cpu_data data)
 
 assert_cpu_is_self(cpu);
 
-tb_lock();
-
 tlb_debug("start: mmu_idx:0x%04lx\n", mmu_idx_bitmask);
 
 for (mmu_idx = 0; mmu_idx < NB_MMU_MODES; mmu_idx++) {
@@ -197,8 +191,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, 
run_on_cpu_data data)
 cpu_tb_jmp_cache_clear(cpu);
 
 tlb_debug("done\n");
-
-tb_unlock();
 }
 
 void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
-- 
2.17.1

Re: [Qemu-devel] [PATCH v2 0/2] kvm: x86 CPU power management

[Marcelo Tosatti]

On Thu, Jun 14, 2018 at 02:37:28AM +0300, Michael S. Tsirkin wrote:
> On Wed, Jun 13, 2018 at 07:34:53PM -0300, Marcelo Tosatti wrote:
> > On Tue, Jun 12, 2018 at 09:47:11PM +0300, Michael S. Tsirkin wrote:
> > > This adds ability to expose host CPU power management capabilities to
> > > guests. For intel guests, this is sufficient for guest to enable
> > > low power CPU power management. For AMD guests it isn't sufficient,
> > > deeper C-states are entered using System-IO.
> > > 
> > > mwait based power management is tied closely to specifics of CPUID,
> > > making migration challenging. At this point only the non-migrateable
> > > -cpu host is supported.
> > > 
> > > With this patch applied, VM latency is within the noise of
> > > baremetal for some benchmarks.
> > > 
> > > perf bench sched pipe results:
> > > Before:
> > > 6.452 sec
> > > After:
> > > 4.382 sec
> > > Baremetal:
> > > 4.136 sec
> > > 
> > > Michael S. Tsirkin (2):
> > >   kvm: support -realtime cpu-pm=on|off
> > >   i386/cpu: make -cpu host support monitor/mwait
> > > 
> > >  include/sysemu/sysemu.h |  1 +
> > >  target/i386/cpu.h   |  9 +
> > >  target/i386/cpu.c   | 19 ++-
> > >  target/i386/kvm.c   | 30 ++
> > >  vl.c|  6 ++
> > >  qemu-options.hx |  9 +++--
> > >  6 files changed, 67 insertions(+), 7 deletions(-)
> > > 
> > > -- 
> > > MST
> > 
> > Hi Michael,
> > 
> > 1) Command line option interface
> > 
> > Why is this not an optional cpu feature such as the other features? 
> > 
> > 
> > -cpu CPU,+mwait  
> > 
> > rather than a separate, architecture independent "-realtime cpu-pm=on|off" 
> > command line option?
> 
> Because it's not just a guest flag. With guest pm on, one guest
> can severely affect the latency of others on the same host CPU.

How so ?

> > 2) Migration
> > 
> > Isnt it sufficient to check that both CPUID leafs are the same, 
> > to allow migration ?
> 
> Not at the moment since linux guests use mwait hints and latency values
> from a table in intel_idle.  If the host and guest models do not match,
> mwait will get a wrong hint.
> 
> It will not do the right thing then!
> 
> You want exactly the same host CPU for it to work.
> 
> This isn't different from how -host cpu works generally.

Ok, makes sense.

> > 1. Check that the processor supports MONITOR and MWAIT. If
> > CPUID.01H:ECX.MONITOR[bit 3] = 1, MONITOR and MWAIT are available at
> > ring 0.
> > 
> > 2. Query the smallest and largest line size that MONITOR uses.
> > Use CPUID.05H:EAX.smallest[bits 15:0];EBX.largest[bits15:0]. 
> > 

[Qemu-devel] [PATCH v7 05/54] docker: Add "cc" subcommand

[Alex Bennée]

Signed-off-by: Fam Zheng 
[AJB: add if args.paths check]
Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v2
  - add if args.paths check to avoid iterating null argument
---
 tests/docker/docker.py | 25 +
 1 file changed, 25 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 306e14cf69..4cd6f9b8ec 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -409,6 +409,31 @@ class ProbeCommand(SubCommand):
 return
 
 
+class CcCommand(SubCommand):
+"""Compile sources with cc in images"""
+name = "cc"
+
+def args(self, parser):
+parser.add_argument("--image", "-i", required=True,
+help="The docker image in which to run cc")
+parser.add_argument("--source-path", "-s", nargs="*", dest="paths",
+help="""Extra paths to (ro) mount into container 
for
+reading sources""")
+
+def run(self, args, argv):
+if argv and argv[0] == "--":
+argv = argv[1:]
+cwd = os.getcwd()
+cmd = ["--rm", "-w", cwd,
+   "-v", "%s:%s:rw" % (cwd, cwd)]
+if args.paths:
+for p in args.paths:
+cmd += ["-v", "%s:%s:ro,z" % (p, p)]
+cmd += [args.image, "cc"]
+cmd += argv
+return Docker().command("run", cmd, args.quiet)
+
+
 def main():
 parser = argparse.ArgumentParser(description="A Docker helper",
 usage="%s  ..." % os.path.basename(sys.argv[0]))
-- 
2.17.1

[Qemu-devel] [PATCH v7 53/54] tests/docker/Makefile.include: only force SID to NOCACHE if old

[Alex Bennée]

Now we can check the age of a docker image we can be a little more
intelligent about re-building Sid images and only force NOCACHE if
it is "old".

Signed-off-by: Alex Bennée 
---
 tests/docker/Makefile.include | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index c0f09505bb..193fcc79fd 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -75,8 +75,14 @@ docker-image-debian-win64-cross: docker-image-debian8-mxe
 # force a re-build of the base image if we ever need to build one of
 # its children.
 ifndef SKIP_DOCKER_BUILD
+ifeq ($(HAVE_USER_DOCKER),y)
+SID_AGE=$(shell $(DOCKER_SCRIPT) check --checktype=age --olderthan=60 --quiet 
qemu:debian-sid)
+ifeq ($(SID_AGE),)
+else
 docker-image-debian-sid: NOCACHE=1
 endif
+endif
+endif
 
 docker-image-debian-alpha-cross: docker-image-debian-sid
 docker-image-debian-hppa-cross: docker-image-debian-sid
-- 
2.17.1

[Qemu-devel] [PULL v2 12/19] translate-all: add page_locked assertions

[Richard Henderson]

From: "Emilio G. Cota" 

This is only compiled under CONFIG_DEBUG_TCG to avoid
bloating the binary.

In user-mode, assert_page_locked is equivalent to assert_mmap_lock.

Note: There are some tb_lock assertions left that will be
removed by later patches.

Reviewed-by: Richard Henderson 
Suggested-by: Alex Bennée 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 accel/tcg/translate-all.c | 82 +--
 1 file changed, 79 insertions(+), 3 deletions(-)

diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 1cc7aab82c..8b378586f4 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -583,6 +583,9 @@ static void page_lock_pair(PageDesc **ret_p1, 
tb_page_addr_t phys1,
 
 /* In user-mode page locks aren't used; mmap_lock is enough */
 #ifdef CONFIG_USER_ONLY
+
+#define assert_page_locked(pd) tcg_debug_assert(have_mmap_lock())
+
 static inline void page_lock(PageDesc *pd)
 { }
 
@@ -605,14 +608,80 @@ void page_collection_unlock(struct page_collection *set)
 { }
 #else /* !CONFIG_USER_ONLY */
 
+#ifdef CONFIG_DEBUG_TCG
+
+static __thread GHashTable *ht_pages_locked_debug;
+
+static void ht_pages_locked_debug_init(void)
+{
+if (ht_pages_locked_debug) {
+return;
+}
+ht_pages_locked_debug = g_hash_table_new(NULL, NULL);
+}
+
+static bool page_is_locked(const PageDesc *pd)
+{
+PageDesc *found;
+
+ht_pages_locked_debug_init();
+found = g_hash_table_lookup(ht_pages_locked_debug, pd);
+return !!found;
+}
+
+static void page_lock__debug(PageDesc *pd)
+{
+ht_pages_locked_debug_init();
+g_assert(!page_is_locked(pd));
+g_hash_table_insert(ht_pages_locked_debug, pd, pd);
+}
+
+static void page_unlock__debug(const PageDesc *pd)
+{
+bool removed;
+
+ht_pages_locked_debug_init();
+g_assert(page_is_locked(pd));
+removed = g_hash_table_remove(ht_pages_locked_debug, pd);
+g_assert(removed);
+}
+
+static void
+do_assert_page_locked(const PageDesc *pd, const char *file, int line)
+{
+if (unlikely(!page_is_locked(pd))) {
+error_report("assert_page_lock: PageDesc %p not locked @ %s:%d",
+ pd, file, line);
+abort();
+}
+}
+
+#define assert_page_locked(pd) do_assert_page_locked(pd, __FILE__, __LINE__)
+
+#else /* !CONFIG_DEBUG_TCG */
+
+#define assert_page_locked(pd)
+
+static inline void page_lock__debug(const PageDesc *pd)
+{
+}
+
+static inline void page_unlock__debug(const PageDesc *pd)
+{
+}
+
+#endif /* CONFIG_DEBUG_TCG */
+
 static inline void page_lock(PageDesc *pd)
 {
+page_lock__debug(pd);
 qemu_spin_lock(>lock);
 }
 
 static inline void page_unlock(PageDesc *pd)
 {
 qemu_spin_unlock(>lock);
+page_unlock__debug(pd);
 }
 
 /* lock the page(s) of a TB in the correct acquisition order */
@@ -658,6 +727,7 @@ static bool page_entry_trylock(struct page_entry *pe)
 if (!busy) {
 g_assert(!pe->locked);
 pe->locked = true;
+page_lock__debug(pe->pd);
 }
 return busy;
 }
@@ -775,6 +845,7 @@ page_collection_lock(tb_page_addr_t start, tb_page_addr_t 
end)
 g_tree_foreach(set->tree, page_entry_unlock, NULL);
 goto retry;
 }
+assert_page_locked(pd);
 PAGE_FOR_EACH_TB(pd, tb, n) {
 if (page_trylock_add(set, tb->page_addr[0]) ||
 (tb->page_addr[1] != -1 &&
@@ -1113,6 +1184,7 @@ static TranslationBlock *tb_alloc(target_ulong pc)
 /* call with @p->lock held */
 static inline void invalidate_page_bitmap(PageDesc *p)
 {
+assert_page_locked(p);
 #ifdef CONFIG_SOFTMMU
 g_free(p->code_bitmap);
 p->code_bitmap = NULL;
@@ -1269,6 +1341,7 @@ static inline void tb_page_remove(PageDesc *pd, 
TranslationBlock *tb)
 uintptr_t *pprev;
 unsigned int n1;
 
+assert_page_locked(pd);
 pprev = >first_tb;
 PAGE_FOR_EACH_TB(pd, tb1, n1) {
 if (tb1 == tb) {
@@ -1417,6 +1490,7 @@ static void build_page_bitmap(PageDesc *p)
 int n, tb_start, tb_end;
 TranslationBlock *tb;
 
+assert_page_locked(p);
 p->code_bitmap = bitmap_new(TARGET_PAGE_SIZE);
 
 PAGE_FOR_EACH_TB(p, tb, n) {
@@ -1450,7 +1524,7 @@ static inline void tb_page_add(PageDesc *p, 
TranslationBlock *tb,
 bool page_already_protected;
 #endif
 
-assert_memory_lock();
+assert_page_locked(p);
 
 tb->page_addr[n] = page_addr;
 tb->page_next[n] = p->first_tb;
@@ -1721,8 +1795,7 @@ tb_invalidate_phys_page_range__locked(struct 
page_collection *pages,
 uint32_t current_flags = 0;
 #endif /* TARGET_HAS_PRECISE_SMC */
 
-assert_memory_lock();
-assert_tb_locked();
+assert_page_locked(p);
 
 #if defined(TARGET_HAS_PRECISE_SMC)
 if (cpu != NULL) {
@@ -1734,6 +1807,7 @@ tb_invalidate_phys_page_range__locked(struct 
page_collection *pages,
 /* XXX: see if in some cases it could be faster to invalidate all
the code */
 PAGE_FOR_EACH_TB(p, tb, n) {
+assert_page_locked(p);
 

[Qemu-devel] [PATCH v7 09/54] tests/tcg: move architecture independent tests into subdir

[Alex Bennée]

We will want to build these for all supported guest architectures so
lets move them all into one place. We also drop test_path at this
point because it needs qemu utils and glib bits which is hard to
support for cross compiling.

Signed-off-by: Alex Bennée 
Reviewed-by: Thomas Huth 
Reviewed-by: Richard Henderson 
---
v2
  - move VPATH and TESTs setup into multiarch/Makefile.target
  - remove moved bits from tests/tcg/Makefile
v3
  - use LDFLAGS+= for testthread linking
v4
  - make myself MAINTAINER for tests/tcg/multiarch
v5
  - split test-mmap into run-test-mmap and run-test-mmap-%
---
 MAINTAINERS|   4 +
 tests/tcg/Makefile |  31 -
 tests/tcg/README   |  10 +-
 tests/tcg/multiarch/Makefile.target|  36 ++
 tests/tcg/multiarch/README |   1 +
 tests/tcg/{ => multiarch}/linux-test.c |   0
 tests/tcg/{ => multiarch}/sha1.c   |   0
 tests/tcg/{ => multiarch}/test-mmap.c  |   0
 tests/tcg/{ => multiarch}/testthread.c |   0
 tests/tcg/test_path.c  | 157 -
 10 files changed, 45 insertions(+), 194 deletions(-)
 create mode 100644 tests/tcg/multiarch/Makefile.target
 create mode 100644 tests/tcg/multiarch/README
 rename tests/tcg/{ => multiarch}/linux-test.c (100%)
 rename tests/tcg/{ => multiarch}/sha1.c (100%)
 rename tests/tcg/{ => multiarch}/test-mmap.c (100%)
 rename tests/tcg/{ => multiarch}/testthread.c (100%)
 delete mode 100644 tests/tcg/test_path.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 0fb5f38f9f..eccd81e8e7 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -303,6 +303,10 @@ F: target/tricore/
 F: hw/tricore/
 F: include/hw/tricore/
 
+Multiarch Linux User Tests
+M: Alex Bennée 
+F: tests/tcg/multiarch/
+
 Guest CPU Cores (KVM):
 --
 
diff --git a/tests/tcg/Makefile b/tests/tcg/Makefile
index 89e3342f3d..e12395117a 100644
--- a/tests/tcg/Makefile
+++ b/tests/tcg/Makefile
@@ -18,12 +18,9 @@ LDFLAGS=
 # also, pi_10.com runs indefinitely
 
 I386_TESTS=hello-i386 \
-  linux-test \
-  testthread \
   sha1-i386 \
   test-i386 \
   test-i386-fprem \
-  test-mmap \
   # runcom
 
 # native i386 compilers sometimes are not biarch.  assume cross-compilers are
@@ -47,8 +44,6 @@ run-%: %
-$(QEMU) ./$*
 
 run-hello-i386: hello-i386
-run-linux-test: linux-test
-run-testthread: testthread
 run-sha1-i386: sha1-i386
 
 run-test-i386: test-i386
@@ -66,11 +61,6 @@ run-test-x86_64: test-x86_64
-$(QEMU_X86_64) test-x86_64 > test-x86_64.out
@if diff -u test-x86_64.ref test-x86_64.out ; then echo "Auto Test OK"; 
fi
 
-run-test-mmap: test-mmap
-   -$(QEMU) ./test-mmap
-   -$(QEMU) -p 8192 ./test-mmap 8192
-   -$(QEMU) -p 16384 ./test-mmap 16384
-   -$(QEMU) -p 32768 ./test-mmap 32768
 
 run-runcom: runcom
-$(QEMU) ./runcom $(SRC_PATH)/tests/pi_10.com
@@ -80,17 +70,10 @@ run-test_path: test_path
 
 # rules to compile tests
 
-test_path: test_path.o
-
-test_path.o: test_path.c
-
 hello-i386: hello-i386.c
$(CC_I386) -nostdlib $(CFLAGS) -static $(LDFLAGS) -o $@ $<
strip $@
 
-testthread: testthread.c
-   $(CC_I386) $(CFLAGS) $(LDFLAGS) -o $@ $< -lpthread
-
 # i386/x86_64 emulation test (test various opcodes) */
 test-i386: test-i386.c test-i386-code16.S test-i386-vm86.S \
test-i386.h test-i386-shift.h test-i386-muldiv.h
@@ -104,28 +87,14 @@ test-x86_64: test-i386.c \
test-i386.h test-i386-shift.h test-i386-muldiv.h
$(CC_X86_64) $(QEMU_INCLUDES) $(CFLAGS) $(LDFLAGS) -o $@ 
$( test-mmap.out, "TEST", \
+   "$< (default) on $(TARGET_NAME)")
+
+# additional page sizes (defined by each architecture adding to EXTRA_RUNS)
+run-test-mmap-%: test-mmap
+   $(call quiet-command, $(QEMU) -p $* $< > test-mmap-$*.out, "TEST", \
+   "$< ($* byte pages) on $(TARGET_NAME)")
diff --git a/tests/tcg/multiarch/README b/tests/tcg/multiarch/README
new file mode 100644
index 00..522c9d2ea3
--- /dev/null
+++ b/tests/tcg/multiarch/README
@@ -0,0 +1 @@
+Multi-architecture linux-user tests
diff --git a/tests/tcg/linux-test.c b/tests/tcg/multiarch/linux-test.c
similarity index 100%
rename from tests/tcg/linux-test.c
rename to tests/tcg/multiarch/linux-test.c
diff --git a/tests/tcg/sha1.c b/tests/tcg/multiarch/sha1.c
similarity index 100%
rename from tests/tcg/sha1.c
rename to tests/tcg/multiarch/sha1.c
diff --git a/tests/tcg/test-mmap.c b/tests/tcg/multiarch/test-mmap.c
similarity index 100%
rename from tests/tcg/test-mmap.c
rename to tests/tcg/multiarch/test-mmap.c
diff --git a/tests/tcg/testthread.c b/tests/tcg/multiarch/testthread.c
similarity index 100%
rename from tests/tcg/testthread.c
rename to tests/tcg/multiarch/testthread.c
diff --git a/tests/tcg/test_path.c b/tests/tcg/test_path.c
deleted file mode 100644
index 1c29bce263..00
--- a/tests/tcg/test_path.c
+++ /dev/null
@@ -1,157 +0,0 @@
-/* Test 

[Qemu-devel] [PATCH v7 04/54] configure: set cross_cc_FOO for host compiler

[Alex Bennée]

We can build tests for the host system with the compiler that we have
selected.

Signed-off-by: Alex Bennée 
Reviewed-by: Richard Henderson 
---
 configure | 14 +-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index b96017b423..069219e9ae 100755
--- a/configure
+++ b/configure
@@ -1431,26 +1431,38 @@ case "$cpu" in
 ppc)
CPU_CFLAGS="-m32"
LDFLAGS="-m32 $LDFLAGS"
+   cross_cc_powerpc=$cc
+   cross_cc_cflags_powerpc=$CPU_CFLAGS
;;
 ppc64)
CPU_CFLAGS="-m64"
LDFLAGS="-m64 $LDFLAGS"
+   cross_cc_ppc64=$cc
+   cross_cc_cflags_ppc64=$CPU_CFLAGS
;;
 sparc)
CPU_CFLAGS="-m32 -mv8plus -mcpu=ultrasparc"
LDFLAGS="-m32 -mv8plus $LDFLAGS"
+   cross_cc_sparc=$cc
+   cross_cc_cflags_sparc=$CPU_CFLAGS
;;
 sparc64)
CPU_CFLAGS="-m64 -mcpu=ultrasparc"
LDFLAGS="-m64 $LDFLAGS"
+   cross_cc_sparc64=$cc
+   cross_cc_cflags_sparc64=$CPU_CFLAGS
;;
 s390)
CPU_CFLAGS="-m31"
LDFLAGS="-m31 $LDFLAGS"
+   cross_cc_s390=$cc
+   cross_cc_cflags_s390=$CPU_CFLAGS
;;
 s390x)
CPU_CFLAGS="-m64"
LDFLAGS="-m64 $LDFLAGS"
+   cross_cc_s390x=$cc
+   cross_cc_cflags_s390x=$CPU_CFLAGS
;;
 i386)
CPU_CFLAGS="-m32"
@@ -1471,7 +1483,7 @@ case "$cpu" in
CPU_CFLAGS="-mx32"
LDFLAGS="-mx32 $LDFLAGS"
cross_cc_i386=$cc
-   cross_cc_cflags_i386="-m32"
+   cross_cc_cflags_i386=$CPU_CFLAGS
;;
 # No special flags required for other host CPUs
 esac
-- 
2.17.1

[Qemu-devel] [PATCH v7 43/54] tests/tcg/Makefile: update to be called from Makefile.target

[Alex Bennée]

This make is now invoked from each individual target make with the
appropriate CC and EXTRA_CFLAGS set for each guest. It then includes
additional Makefile.targets from:

  - tests/tcg/multiarch (always)
  - tests/tcg/$(TARGET_BASE_ARCH) (if available)
  - tests/tcg/$(TARGET_NAME)

The order is important as the later Makefile's may want to suppress
TESTS from its base arch profile. Each included Makefile.target is
responsible for adding TESTS as well as defining any special build
instructions for individual tests.

Signed-off-by: Alex Bennée 
Reviewed-by: Richard Henderson 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v2
  - cleaner approach to include sub makefiles
  - move TESTS/VPATH manipulation into sub-makefile
  - avoid double inclusion when TARGET_BASE_ARCH==TARGET_NAME
v4
  - add timeout to default runner
  - clean-up comments about build flags
  - update to handle BUILD_STATIC
  - add MAINTAINERS entry
v5
  - support EXTRA_RUNS for run variants
v6
  - extend timeout for DEBUG_TCG
---
 MAINTAINERS|   5 ++
 tests/tcg/Makefile | 183 +++--
 2 files changed, 81 insertions(+), 107 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index c4fc6fa357..3795a32c27 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2150,6 +2150,11 @@ W: https://travis-ci.org/qemu/qemu
 W: https://app.shippable.com/github/qemu/qemu
 W: http://patchew.org/QEMU/
 
+Guest Test Compilation Support
+M: Alex Bennée 
+F: tests/tcg/Makefile
+L: qemu-devel@nongnu.org
+
 Documentation
 -
 Build system architecture
diff --git a/tests/tcg/Makefile b/tests/tcg/Makefile
index e12395117a..e7dbcdb5bf 100644
--- a/tests/tcg/Makefile
+++ b/tests/tcg/Makefile
@@ -1,125 +1,94 @@
+# -*- Mode: makefile -*-
+#
+# TCG tests
+#
+# These are complicated by the fact we want to build them for guest
+# systems. This requires knowing what guests we are building and which
+# ones we have cross-compilers for or docker images with
+# cross-compilers.
+#
+# The tests themselves should be as minimal as possible as
+# cross-compilers don't always have a large amount of libraries
+# available.
+#
+# We only include the host build system for SRC_PATH and we don't
+# bother with the common rules.mk. We expect the following:
+#
+#   CC - the C compiler command
+#   EXTRA_CFLAGS - any extra CFLAGS
+#   BUILD_STATIC - are we building static binaries
+#
+# By default all tests are statically compiled but some host systems
+# may not package static libraries by default. If an external
+# cross-compiler can only build dynamic libraries the user might need
+# to make extra efforts to ensure ld.so can link at runtime when the
+# tests are run.
+#
+# We also accept SPEED=slow to enable slower running tests
+#
+# We also expect to be in the tests build dir for the FOO-linux-user.
+#
+
 -include ../../config-host.mak
--include $(SRC_PATH)/rules.mak
+-include ../config-target.mak
 
-$(call set-vpath, $(SRC_PATH)/tests/tcg)
+quiet-command = $(if $(V),$1,$(if $(2),@printf "  %-7s %s\n" $2 $3 && $1, @$1))
 
-QEMU=../../i386-linux-user/qemu-i386
-QEMU_X86_64=../../x86_64-linux-user/qemu-x86_64
-CC_X86_64=$(CC_I386) -m64
+# Tests we are building
+TESTS=
 
-QEMU_INCLUDES += -I../..
-CFLAGS=-Wall -O2 -g -fno-strict-aliasing
-#CFLAGS+=-msse2
+# Start with a blank slate, the build targets get to add stuff first
+CFLAGS=
+QEMU_CFLAGS=
 LDFLAGS=
 
-# TODO: automatically detect ARM and MIPS compilers, and run those too
-
-# runcom maps page 0, so it requires root privileges
-# also, pi_10.com runs indefinitely
-
-I386_TESTS=hello-i386 \
-  sha1-i386 \
-  test-i386 \
-  test-i386-fprem \
-  # runcom
+# The QEMU for this TARGET
+QEMU=../qemu-$(TARGET_NAME)
 
-# native i386 compilers sometimes are not biarch.  assume cross-compilers are
-ifneq ($(ARCH),i386)
-I386_TESTS+=run-test-x86_64
+# If TCG debugging is enabled things are a lot slower
+ifeq ($(CONFIG_DEBUG_TCG),y)
+TIMEOUT=45
+else
+TIMEOUT=15
 endif
 
-TESTS = test_path
-ifneq ($(call find-in-path, $(CC_I386)),)
-TESTS += $(I386_TESTS)
+# The order we include is important. We include multiarch, base arch
+# and finally arch if it's not the same as base arch.
+-include $(SRC_PATH)/tests/tcg/multiarch/Makefile.target
+-include $(SRC_PATH)/tests/tcg/$(TARGET_BASE_ARCH)/Makefile.target
+ifneq ($(TARGET_BASE_ARCH),$(TARGET_NAME))
+-include $(SRC_PATH)/tests/tcg/$(TARGET_NAME)/Makefile.target
 endif
 
-all: $(patsubst %,run-%,$(TESTS))
-test: all
-
-# rules to run tests
-
-.PHONY: $(patsubst %,run-%,$(TESTS))
-
-run-%: %
-   -$(QEMU) ./$*
-
-run-hello-i386: hello-i386
-run-sha1-i386: sha1-i386
-
-run-test-i386: test-i386
-   ./test-i386 > test-i386.ref
-   -$(QEMU) test-i386 > test-i386.out
-   @if diff -u test-i386.ref test-i386.out ; then echo "Auto Test OK"; fi
-
-run-test-i386-fprem: test-i386-fprem
-   ./test-i386-fprem > test-i386-fprem.ref
-   -$(QEMU) test-i386-fprem > test-i386-fprem.out

[Qemu-devel] [PATCH v7 15/54] tests/tcg: enable building for i386

[Alex Bennée]

While you can construct a compile command that does work using the
x86_64 host compiler that most people use this is flakey. Different
distros handle this is different ways so we default to using a known
good i386 compiler via docker.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 

---
v3
  - add runner for runcom
v4
  - runcom pi10.com directly from SRC_DIR
v5
  - define extra test-mmap runs
---
 tests/tcg/i386/Makefile.include | 9 +
 tests/tcg/i386/Makefile.target  | 7 +++
 2 files changed, 16 insertions(+)
 create mode 100644 tests/tcg/i386/Makefile.include

diff --git a/tests/tcg/i386/Makefile.include b/tests/tcg/i386/Makefile.include
new file mode 100644
index 00..be1c3008dd
--- /dev/null
+++ b/tests/tcg/i386/Makefile.include
@@ -0,0 +1,9 @@
+#
+# Makefile.include for all i386
+#
+# There is enough brokeness in x86_64 compilers that we don't default
+# to using the x86_64 system compiler for i386 binaries.
+#
+
+DOCKER_IMAGE=fedora-i386-cross
+DOCKER_CROSS_COMPILER=gcc
diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target
index 2f27b65e2d..8b46cc021e 100644
--- a/tests/tcg/i386/Makefile.target
+++ b/tests/tcg/i386/Makefile.target
@@ -28,3 +28,10 @@ hello-i386: LDFLAGS+=-nostdlib
 test-i386: test-i386.c test-i386-code16.S test-i386-vm86.S test-i386.h 
test-i386-shift.h test-i386-muldiv.h
$(CC) $(CFLAGS) $(LDFLAGS) -o $@ \
   $( 
runcom.out, "TEST", "$< on $(TARGET_NAME)")
+
+# On i386 and x86_64 Linux only supports 4k pages (large pages are a different 
hack)
+EXTRA_RUNS+=run-test-mmap-4096
-- 
2.17.1

[Qemu-devel] [PULL v2 11/19] translate-all: use per-page locking in !user-mode

[Richard Henderson]

From: "Emilio G. Cota" 

Groundwork for supporting parallel TCG generation.

Instead of using a global lock (tb_lock) to protect changes
to pages, use fine-grained, per-page locks in !user-mode.
User-mode stays with mmap_lock.

Sometimes changes need to happen atomically on more than one
page (e.g. when a TB that spans across two pages is
added/invalidated, or when a range of pages is invalidated).
We therefore introduce struct page_collection, which helps
us keep track of a set of pages that have been locked in
the appropriate locking order (i.e. by ascending page index).

This commit first introduces the structs and the function helpers,
to then convert the calling code to use per-page locking. Note
that tb_lock is not removed yet.

While at it, rename tb_alloc_page to tb_page_add, which pairs with
tb_page_remove.

Reviewed-by: Richard Henderson 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 accel/tcg/translate-all.h |   3 +
 include/exec/exec-all.h   |   3 +-
 accel/tcg/translate-all.c | 444 ++
 3 files changed, 409 insertions(+), 41 deletions(-)

diff --git a/accel/tcg/translate-all.h b/accel/tcg/translate-all.h
index ba8e4d63c4..6d1d2588b5 100644
--- a/accel/tcg/translate-all.h
+++ b/accel/tcg/translate-all.h
@@ -23,6 +23,9 @@
 
 
 /* translate-all.c */
+struct page_collection *page_collection_lock(tb_page_addr_t start,
+ tb_page_addr_t end);
+void page_collection_unlock(struct page_collection *set);
 void tb_invalidate_phys_page_fast(tb_page_addr_t start, int len);
 void tb_invalidate_phys_page_range(tb_page_addr_t start, tb_page_addr_t end,
int is_cpu_write_access);
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 07653d3c92..8d92e3cea9 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -359,7 +359,8 @@ struct TranslationBlock {
 /* original tb when cflags has CF_NOCACHE */
 struct TranslationBlock *orig_tb;
 /* first and second physical page containing code. The lower bit
-   of the pointer tells the index in page_next[] */
+   of the pointer tells the index in page_next[].
+   The list is protected by the TB's page('s) lock(s) */
 uintptr_t page_next[2];
 tb_page_addr_t page_addr[2];
 
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 83bb40fb20..1cc7aab82c 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -112,8 +112,55 @@ typedef struct PageDesc {
 #else
 unsigned long flags;
 #endif
+#ifndef CONFIG_USER_ONLY
+QemuSpin lock;
+#endif
 } PageDesc;
 
+/**
+ * struct page_entry - page descriptor entry
+ * @pd: pointer to the  PageDesc of the page this entry represents
+ * @index:  page index of the page
+ * @locked: whether the page is locked
+ *
+ * This struct helps us keep track of the locked state of a page, without
+ * bloating  PageDesc.
+ *
+ * A page lock protects accesses to all fields of  PageDesc.
+ *
+ * See also:  page_collection.
+ */
+struct page_entry {
+PageDesc *pd;
+tb_page_addr_t index;
+bool locked;
+};
+
+/**
+ * struct page_collection - tracks a set of pages (i.e.  page_entry's)
+ * @tree:   Binary search tree (BST) of the pages, with key == page index
+ * @max:Pointer to the page in @tree with the highest page index
+ *
+ * To avoid deadlock we lock pages in ascending order of page index.
+ * When operating on a set of pages, we need to keep track of them so that
+ * we can lock them in order and also unlock them later. For this we collect
+ * pages (i.e.  page_entry's) in a binary search @tree. Given that the
+ * @tree implementation we use does not provide an O(1) operation to obtain the
+ * highest-ranked element, we use @max to keep track of the inserted page
+ * with the highest index. This is valuable because if a page is not in
+ * the tree and its index is higher than @max's, then we can lock it
+ * without breaking the locking order rule.
+ *
+ * Note on naming: 'struct page_set' would be shorter, but we already have a 
few
+ * page_set_*() helpers, so page_collection is used instead to avoid confusion.
+ *
+ * See also: page_collection_lock().
+ */
+struct page_collection {
+GTree *tree;
+struct page_entry *max;
+};
+
 /* list iterators for lists of tagged pointers in TranslationBlock */
 #define TB_FOR_EACH_TAGGED(head, tb, n, field)  \
 for (n = (head) & 1, tb = (TranslationBlock *)((head) & ~1);\
@@ -507,6 +554,15 @@ static PageDesc *page_find_alloc(tb_page_addr_t index, int 
alloc)
 return NULL;
 }
 pd = g_new0(PageDesc, V_L2_SIZE);
+#ifndef CONFIG_USER_ONLY
+{
+int i;
+
+for (i = 0; i < V_L2_SIZE; i++) {
+qemu_spin_init([i].lock);
+}
+}
+#endif
 existing = atomic_cmpxchg(lp, NULL, pd);
 if (unlikely(existing)) {
 

[Qemu-devel] [PATCH v7 03/54] configure: allow user to specify --cross-cc-cflags-foo=

[Alex Bennée]

As an individual compiler may be able to support several targets with
the appropriate flags we need to expose this to the user as well.

Signed-off-by: Alex Bennée 
Reviewed-by: Richard Henderson 
---
v3
  - fix up the cross-cc
v4
  - add --help text
  - sort cross_cc_foo lines
---
 configure | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/configure b/configure
index 6e33dbbe10..b96017b423 100755
--- a/configure
+++ b/configure
@@ -459,10 +459,14 @@ docker="no"
 
 # cross compilers defaults, can be overridden with --cross-cc-ARCH
 cross_cc_aarch64="aarch64-linux-gnu-gcc"
+cross_cc_aarch64_be="$cross_cc_aarch64"
+cross_cc_cflags_aarch64_be="-mbig-endian"
 cross_cc_arm="arm-linux-gnueabihf-gcc"
+cross_cc_cflags_armeb="-mbig-endian"
 cross_cc_i386="i386-pc-linux-gnu-gcc"
 cross_cc_cflags_i386=""
 cross_cc_powerpc="powerpc-linux-gnu-gcc"
+cross_cc_powerpc="powerpc-linux-gnu-gcc"
 
 enabled_cross_compilers=""
 
@@ -498,6 +502,9 @@ for opt do
   ;;
   --cross-cc-*[!a-zA-Z0-9_-]*=*) error_exit "Passed bad --cross-cc-FOO option"
   ;;
+  --cross-cc-cflags-*) cc_arch=${opt#--cross-cc-flags-}; cc_arch=${cc_arch%%=*}
+  eval "cross_cc_cflags_${cc_arch}=\$optarg"
+  ;;
   --cross-cc-*) cc_arch=${opt#--cross-cc-}; cc_arch=${cc_arch%%=*}
 eval "cross_cc_${cc_arch}=\$optarg"
   ;;
@@ -1527,6 +1534,7 @@ Advanced options (experts only):
   --extra-cxxflags=CXXFLAGS append extra C++ compiler flags QEMU_CXXFLAGS
   --extra-ldflags=LDFLAGS  append extra linker flags LDFLAGS
   --cross-cc-ARCH=CC   use compiler when building ARCH guest test cases
+  --cross-cc-flags-ARCH=   use compiler flags when building ARCH guest tests
   --make=MAKE  use specified make [$make]
   --install=INSTALLuse specified install [$install]
   --python=PYTHON  use specified python [$python]
@@ -6882,6 +6890,7 @@ case "$target_name" in
 mttcg="yes"
 gdb_xml_files="arm-core.xml arm-vfp.xml arm-vfp3.xml arm-neon.xml"
 target_compiler=$cross_cc_arm
+eval "target_compiler_cflags=\$cross_cc_cflags_${target_name}"
   ;;
   aarch64|aarch64_be)
 TARGET_ARCH=aarch64
@@ -6890,6 +6899,7 @@ case "$target_name" in
 mttcg="yes"
 gdb_xml_files="aarch64-core.xml aarch64-fpu.xml arm-core.xml arm-vfp.xml 
arm-vfp3.xml arm-neon.xml"
 target_compiler=$cross_cc_aarch64
+eval "target_compiler_cflags=\$cross_cc_cflags_${target_name}"
   ;;
   cris)
 target_compiler=$cross_cc_cris
-- 
2.17.1

[Qemu-devel] [PATCH v7 08/54] docker: Makefile.include introduce DOCKER_SCRIPT

[Alex Bennée]

Define this in one place to make it easy to re-use.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
---
 tests/docker/Makefile.include | 10 ++
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 74fd51c22c..8afb383478 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -15,6 +15,8 @@ DOCKER_TESTS := $(notdir $(shell \
 
 DOCKER_TOOLS := travis
 
+DOCKER_SCRIPT=$(SRC_PATH)/tests/docker/docker.py
+
 TESTS ?= %
 IMAGES ?= %
 
@@ -38,7 +40,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
echo WARNING: EXECUTABLE is not set, debootstrap may fail. 2>&1 
; \
fi
$(call quiet-command,\
-   $(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
+   $(DOCKER_SCRIPT) build qemu:$* $< \
$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
$(if $(NOUSER),,--add-current-user) \
$(if $(EXTRA_FILES),--extra-files $(EXTRA_FILES))\
@@ -133,11 +135,11 @@ docker-run: docker-qemu-src
fi
$(if $(EXECUTABLE), \
$(call quiet-command,   \
-   $(SRC_PATH)/tests/docker/docker.py update   \
+   $(DOCKER_SCRIPT) update \
$(IMAGE) $(EXECUTABLE), \
"  COPYING $(EXECUTABLE) to $(IMAGE)"))
$(call quiet-command,   \
-   $(SRC_PATH)/tests/docker/docker.py run  \
+   $(DOCKER_SCRIPT) run\
$(if $(NOUSER),,-u $(shell id -u))  \
--security-opt seccomp=unconfined   \
$(if $V,,--rm)  \
@@ -167,4 +169,4 @@ docker-run-%:
@$(MAKE) docker-run TEST=$(CMD) IMAGE=qemu:$(IMAGE)
 
 docker-clean:
-   $(call quiet-command, $(SRC_PATH)/tests/docker/docker.py clean)
+   $(call quiet-command, $(DOCKER_SCRIPT) clean)
-- 
2.17.1

[Qemu-devel] [PULL v2 10/19] translate-all: move tb_invalidate_phys_page_range up in the file

[Richard Henderson]

From: "Emilio G. Cota" 

This greatly simplifies next commit's diff.

Reviewed-by: Richard Henderson 
Reviewed-by: Alex Bennée 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 accel/tcg/translate-all.c | 77 ---
 1 file changed, 39 insertions(+), 38 deletions(-)

diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index f32904d4a3..83bb40fb20 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -1363,44 +1363,6 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
 return tb;
 }
 
-/*
- * Invalidate all TBs which intersect with the target physical address range
- * [start;end[. NOTE: start and end may refer to *different* physical pages.
- * 'is_cpu_write_access' should be true if called from a real cpu write
- * access: the virtual CPU will exit the current TB if code is modified inside
- * this TB.
- *
- * Called with mmap_lock held for user-mode emulation, grabs tb_lock
- * Called with tb_lock held for system-mode emulation
- */
-static void tb_invalidate_phys_range_1(tb_page_addr_t start, tb_page_addr_t 
end)
-{
-tb_page_addr_t next;
-
-for (next = (start & TARGET_PAGE_MASK) + TARGET_PAGE_SIZE;
- start < end;
- start = next, next += TARGET_PAGE_SIZE) {
-tb_page_addr_t bound = MIN(next, end);
-
-tb_invalidate_phys_page_range(start, bound, 0);
-}
-}
-
-#ifdef CONFIG_SOFTMMU
-void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end)
-{
-assert_tb_locked();
-tb_invalidate_phys_range_1(start, end);
-}
-#else
-void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end)
-{
-assert_memory_lock();
-tb_lock();
-tb_invalidate_phys_range_1(start, end);
-tb_unlock();
-}
-#endif
 /*
  * Invalidate all TBs which intersect with the target physical address range
  * [start;end[. NOTE: start and end must refer to the *same* physical page.
@@ -1500,6 +1462,45 @@ void tb_invalidate_phys_page_range(tb_page_addr_t start, 
tb_page_addr_t end,
 #endif
 }
 
+/*
+ * Invalidate all TBs which intersect with the target physical address range
+ * [start;end[. NOTE: start and end may refer to *different* physical pages.
+ * 'is_cpu_write_access' should be true if called from a real cpu write
+ * access: the virtual CPU will exit the current TB if code is modified inside
+ * this TB.
+ *
+ * Called with mmap_lock held for user-mode emulation, grabs tb_lock
+ * Called with tb_lock held for system-mode emulation
+ */
+static void tb_invalidate_phys_range_1(tb_page_addr_t start, tb_page_addr_t 
end)
+{
+tb_page_addr_t next;
+
+for (next = (start & TARGET_PAGE_MASK) + TARGET_PAGE_SIZE;
+ start < end;
+ start = next, next += TARGET_PAGE_SIZE) {
+tb_page_addr_t bound = MIN(next, end);
+
+tb_invalidate_phys_page_range(start, bound, 0);
+}
+}
+
+#ifdef CONFIG_SOFTMMU
+void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end)
+{
+assert_tb_locked();
+tb_invalidate_phys_range_1(start, end);
+}
+#else
+void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end)
+{
+assert_memory_lock();
+tb_lock();
+tb_invalidate_phys_range_1(start, end);
+tb_unlock();
+}
+#endif
+
 #ifdef CONFIG_SOFTMMU
 /* len must be <= 8 and start must be a multiple of len.
  * Called via softmmu_template.h when code areas are written to with
-- 
2.17.1

[Qemu-devel] [PATCH v7 23/54] tests/tcg: enable building for ARM

[Alex Bennée]

This allows us to use the docker cross compiler image to build these
tests.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
v5
  - add EXTRA_RUNS for mmap tests
---
 tests/tcg/arm/Makefile.include | 8 
 tests/tcg/arm/Makefile.target  | 5 -
 2 files changed, 12 insertions(+), 1 deletion(-)
 create mode 100644 tests/tcg/arm/Makefile.include

diff --git a/tests/tcg/arm/Makefile.include b/tests/tcg/arm/Makefile.include
new file mode 100644
index 00..8e7eac008f
--- /dev/null
+++ b/tests/tcg/arm/Makefile.include
@@ -0,0 +1,8 @@
+# Makefile.include for all ARM targets
+#
+# We don't have any bigendian build tools so we only use this for armhf
+
+ifeq ($(TARGET_NAME),arm)
+DOCKER_IMAGE=debian-armhf-cross
+DOCKER_CROSS_COMPILER=arm-linux-gnueabihf-gcc
+endif
diff --git a/tests/tcg/arm/Makefile.target b/tests/tcg/arm/Makefile.target
index bc6962ecc6..0312293dca 100644
--- a/tests/tcg/arm/Makefile.target
+++ b/tests/tcg/arm/Makefile.target
@@ -1,6 +1,6 @@
 # -*- Mode: makefile -*-
 #
-# ARM - included from tests/tcg/Makefile.target
+# ARM - included from tests/tcg/Makefile
 #
 
 ARM_SRC=$(SRC_PATH)/tests/tcg/arm
@@ -11,3 +11,6 @@ VPATH += $(ARM_SRC)
 
 hello-arm: CFLAGS+=-marm -ffreestanding
 hello-arm: LDFLAGS+=-nostdlib
+
+# On ARM Linux only supports 4k pages
+EXTRA_RUNS+=run-test-mmap-4096
-- 
2.17.1

[Qemu-devel] [PATCH v7 06/54] docker: extend "cc" command to accept compiler

[Alex Bennée]

When calling our cross-compilation images we want to call something
other than the default cc.

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Reviewed-by: Fam Zheng 
---
v2
  - use arg.cc default to simplify logic
---
 tests/docker/docker.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 4cd6f9b8ec..663dfcba4b 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -416,6 +416,8 @@ class CcCommand(SubCommand):
 def args(self, parser):
 parser.add_argument("--image", "-i", required=True,
 help="The docker image in which to run cc")
+parser.add_argument("--cc", default="cc",
+help="The compiler executable to call")
 parser.add_argument("--source-path", "-s", nargs="*", dest="paths",
 help="""Extra paths to (ro) mount into container 
for
 reading sources""")
@@ -429,7 +431,7 @@ class CcCommand(SubCommand):
 if args.paths:
 for p in args.paths:
 cmd += ["-v", "%s:%s:ro,z" % (p, p)]
-cmd += [args.image, "cc"]
+cmd += [args.image, args.cc]
 cmd += argv
 return Docker().command("run", cmd, args.quiet)
 
-- 
2.17.1

[Qemu-devel] [PULL v2 09/19] translate-all: work page-by-page in tb_invalidate_phys_range_1

[Richard Henderson]

From: "Emilio G. Cota" 

So that we pass a same-page range to tb_invalidate_phys_page_range,
instead of always passing an end address that could be on a different
page.

As discussed with Peter Maydell on the list [1], tb_invalidate_phys_page_range
doesn't actually do much with 'end', which explains why we have never
hit a bug despite going against what the comment on top of
tb_invalidate_phys_page_range requires:

> * Invalidate all TBs which intersect with the target physical address range
> * [start;end[. NOTE: start and end must refer to the *same* physical page.

The appended honours the comment, which avoids confusion.

While at it, rework the loop into a for loop, which is less error prone
(e.g. "continue" won't result in an infinite loop).

[1] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg09165.html

Reviewed-by: Richard Henderson 
Reviewed-by: Alex Bennée 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 accel/tcg/translate-all.c | 12 
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index b9c36a3e45..f32904d4a3 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -1375,10 +1375,14 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
  */
 static void tb_invalidate_phys_range_1(tb_page_addr_t start, tb_page_addr_t 
end)
 {
-while (start < end) {
-tb_invalidate_phys_page_range(start, end, 0);
-start &= TARGET_PAGE_MASK;
-start += TARGET_PAGE_SIZE;
+tb_page_addr_t next;
+
+for (next = (start & TARGET_PAGE_MASK) + TARGET_PAGE_SIZE;
+ start < end;
+ start = next, next += TARGET_PAGE_SIZE) {
+tb_page_addr_t bound = MIN(next, end);
+
+tb_invalidate_phys_page_range(start, bound, 0);
 }
 }
 
-- 
2.17.1

[Qemu-devel] [PATCH v7 07/54] docker: allow "cc" command to run in user context

[Alex Bennée]

Signed-off-by: Alex Bennée 
Reviewed-by: Fam Zheng 
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
---
 tests/docker/docker.py | 4 
 1 file changed, 4 insertions(+)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 663dfcba4b..c83fd89985 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -418,6 +418,8 @@ class CcCommand(SubCommand):
 help="The docker image in which to run cc")
 parser.add_argument("--cc", default="cc",
 help="The compiler executable to call")
+parser.add_argument("--user",
+help="The user-id to run under")
 parser.add_argument("--source-path", "-s", nargs="*", dest="paths",
 help="""Extra paths to (ro) mount into container 
for
 reading sources""")
@@ -431,6 +433,8 @@ class CcCommand(SubCommand):
 if args.paths:
 for p in args.paths:
 cmd += ["-v", "%s:%s:ro,z" % (p, p)]
+if args.user:
+cmd += ["-u", args.user]
 cmd += [args.image, args.cc]
 cmd += argv
 return Docker().command("run", cmd, args.quiet)
-- 
2.17.1

[Qemu-devel] [PULL v2 19/19] tcg: Reduce max TB opcode count

[Richard Henderson]

Also, assert that we don't overflow any of two different offsets into
the TB. Both unwind and goto_tb both record a uint16_t for later use.

This fixes an arm-softmmu test case utilizing NEON in which there is
a TB generated that runs to 7800 opcodes, and compiles to 96k on an
x86_64 host.  This overflows the 16-bit offset in which we record the
goto_tb reset offset.  Because of that overflow, we install a jump
destination that goes to neverland.  Boom.

With this reduced op count, the same TB compiles to about 48k for
aarch64, ppc64le, and x86_64 hosts, and neither assertion fires.

Cc: qemu-sta...@nongnu.org
Reported-by: "Jason A. Donenfeld" 
Reviewed-by: Philippe Mathieu-Daudé 
Signed-off-by: Richard Henderson 
---
 tcg/tcg.h|  6 --
 tcg/aarch64/tcg-target.inc.c |  2 +-
 tcg/arm/tcg-target.inc.c |  2 +-
 tcg/i386/tcg-target.inc.c|  2 +-
 tcg/mips/tcg-target.inc.c|  2 +-
 tcg/ppc/tcg-target.inc.c |  4 ++--
 tcg/s390/tcg-target.inc.c|  2 +-
 tcg/sparc/tcg-target.inc.c   |  4 ++--
 tcg/tcg.c| 13 -
 tcg/tci/tcg-target.inc.c |  2 +-
 10 files changed, 26 insertions(+), 13 deletions(-)

diff --git a/tcg/tcg.h b/tcg/tcg.h
index 532d2a0710..f9f12378e9 100644
--- a/tcg/tcg.h
+++ b/tcg/tcg.h
@@ -850,9 +850,11 @@ static inline bool tcg_op_buf_full(void)
 /* This is not a hard limit, it merely stops translation when
  * we have produced "enough" opcodes.  We want to limit TB size
  * such that a RISC host can reasonably use a 16-bit signed
- * branch within the TB.
+ * branch within the TB.  We also need to be mindful of the
+ * 16-bit unsigned offsets, TranslationBlock.jmp_reset_offset[]
+ * and TCGContext.gen_insn_end_off[].
  */
-return tcg_ctx->nb_ops >= 8000;
+return tcg_ctx->nb_ops >= 4000;
 }
 
 /* pool based memory allocation */
diff --git a/tcg/aarch64/tcg-target.inc.c b/tcg/aarch64/tcg-target.inc.c
index be3192078d..4562d36d1b 100644
--- a/tcg/aarch64/tcg-target.inc.c
+++ b/tcg/aarch64/tcg-target.inc.c
@@ -1733,7 +1733,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
 tcg_out_insn(s, 3305, LDR, offset, TCG_REG_TMP);
 }
 tcg_out_insn(s, 3207, BR, TCG_REG_TMP);
-s->tb_jmp_reset_offset[a0] = tcg_current_code_size(s);
+set_jmp_reset_offset(s, a0);
 break;
 
 case INDEX_op_goto_ptr:
diff --git a/tcg/arm/tcg-target.inc.c b/tcg/arm/tcg-target.inc.c
index 56a32a470f..e1fbf465cb 100644
--- a/tcg/arm/tcg-target.inc.c
+++ b/tcg/arm/tcg-target.inc.c
@@ -1822,7 +1822,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode 
opc,
 tcg_out_movi32(s, COND_AL, base, ptr - dil);
 }
 tcg_out_ld32_12(s, COND_AL, TCG_REG_PC, base, dil);
-s->tb_jmp_reset_offset[args[0]] = tcg_current_code_size(s);
+set_jmp_reset_offset(s, args[0]);
 }
 break;
 case INDEX_op_goto_ptr:
diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c
index 0d0ff524b7..e87b0d445e 100644
--- a/tcg/i386/tcg-target.inc.c
+++ b/tcg/i386/tcg-target.inc.c
@@ -2245,7 +2245,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode 
opc,
 tcg_out_modrm_offset(s, OPC_GRP5, EXT5_JMPN_Ev, -1,
  (intptr_t)(s->tb_jmp_target_addr + a0));
 }
-s->tb_jmp_reset_offset[a0] = tcg_current_code_size(s);
+set_jmp_reset_offset(s, a0);
 break;
 case INDEX_op_goto_ptr:
 /* jmp to the given host address (could be epilogue) */
diff --git a/tcg/mips/tcg-target.inc.c b/tcg/mips/tcg-target.inc.c
index ca5f1d4894..cff525373b 100644
--- a/tcg/mips/tcg-target.inc.c
+++ b/tcg/mips/tcg-target.inc.c
@@ -1744,7 +1744,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode 
opc,
 tcg_out_opc_reg(s, OPC_JR, 0, TCG_TMP0, 0);
 }
 tcg_out_nop(s);
-s->tb_jmp_reset_offset[a0] = tcg_current_code_size(s);
+set_jmp_reset_offset(s, a0);
 break;
 case INDEX_op_goto_ptr:
 /* jmp to the given host address (could be epilogue) */
diff --git a/tcg/ppc/tcg-target.inc.c b/tcg/ppc/tcg-target.inc.c
index 86f7de5f7e..c2f729ee8f 100644
--- a/tcg/ppc/tcg-target.inc.c
+++ b/tcg/ppc/tcg-target.inc.c
@@ -2025,10 +2025,10 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, 
const TCGArg *args,
 }
 tcg_out32(s, MTSPR | RS(TCG_REG_TB) | CTR);
 tcg_out32(s, BCCTR | BO_ALWAYS);
-s->tb_jmp_reset_offset[args[0]] = c = tcg_current_code_size(s);
+set_jmp_reset_offset(s, args[0]);
 if (USE_REG_TB) {
 /* For the unlinked case, need to reset TCG_REG_TB.  */
-c = -c;
+c = -tcg_current_code_size(s);
 assert(c == (int16_t)c);
 tcg_out32(s, ADDI | TAI(TCG_REG_TB, TCG_REG_TB, c));
 }
diff --git a/tcg/s390/tcg-target.inc.c b/tcg/s390/tcg-target.inc.c
index 9af6dcef05..17c435ade5 100644

[Qemu-devel] [PULL v2 18/19] tcg: remove tb_lock

[Richard Henderson]

From: "Emilio G. Cota" 

Use mmap_lock in user-mode to protect TCG state and the page descriptors.
In !user-mode, each vCPU has its own TCG state, so no locks needed.
Per-page locks are used to protect the page descriptors.

Per-TB locks are used in both modes to protect TB jumps.

Some notes:

- tb_lock is removed from notdirty_mem_write by passing a
  locked page_collection to tb_invalidate_phys_page_fast.

- tcg_tb_lookup/remove/insert/etc have their own internal lock(s),
  so there is no need to further serialize access to them.

- do_tb_flush is run in a safe async context, meaning no other
  vCPU threads are running. Therefore acquiring mmap_lock there
  is just to please tools such as thread sanitizer.

- Not visible in the diff, but tb_invalidate_phys_page already
  has an assert_memory_lock.

- cpu_io_recompile is !user-only, so no mmap_lock there.

- Added mmap_unlock()'s before all siglongjmp's that could
  be called in user-mode while mmap_lock is held.
  + Added an assert for !have_mmap_lock() after returning from
the longjmp in cpu_exec, just like we do in cpu_exec_step_atomic.

Performance numbers before/after:

Host: AMD Opteron(tm) Processor 6376

 ubuntu 17.04 ppc64 bootup+shutdown time

  700 +-+--++--++---+*--+-+
  |++  ++   +   *B|
  | before ***B***** *|
  |tb lock removal ###D### ***|
  600 +-+   *** +-+
  |   ** #|
  |*B*  #D|
  | *** * ##  |
  500 +-+***   ###  +-+
  | * ***   ###   |
  |*B*  # ##  |
  |  ** *  #D#|
  400 +-+  **## +-+
  |  **   ### |
  |**   ##|
  |  ** # ##  |
  300 +-+  *   B*  #D#  +-+
  |B ***###   |
  |*   **     |
  | *   ***  ###  |
  200 +-+   B  *B #D#   +-+
  | #B* *   ## #  |
  | #*##  |
  |+ D##D# ++   ++|
  100 +-+--++--++---++--+-+
   18  16  Guest CPUs   48   64
  png: https://imgur.com/HwmBHXe

  debian jessie aarch64 bootup+shutdown time

  90 +-+--+-+-++++--+-+
 |+ + ++++|
 | before ***B***B|
  80 +tb lock removal ###D###  **D  +-+
 |   **###|
 | **##   |
  70 +-+ ** #   +-+
 | ** ##  |
 |   **  #|
  60 +-+   *B  ##   +-+
 |   **  ##   |
 |***  #D |
  50 +-+   ***   ## +-+
 | * **   ### |
 |   **B*  ###|
  40 +-+   # ## +-+
 |    #D# |
 | ***B**  ###|
  30 +-+B***B** +-+
 |B *   * # ###   |
 | B   ###D#  |
  20 +-+   D  ##D## +-+
 |  D#|
 |+ + ++++|
  10 +-+--+-+-++++--+-+
  1 8 16  Guest CPUs48   64
  png: https://imgur.com/iGpGFtv

The gains are high for 4-8 CPUs. Beyond that point, however, unrelated
lock contention 

[Qemu-devel] [PATCH v7 00/54] fix building of tests/tcg - last chance to review!

[Alex Bennée]

Hi,

Most of the work has been trying to get make check-tcg to work
reliably under high -j's. This has led to a couple of fixes to actual
tests (see tests/tcg/multiarch: don't hard code paths/ports for
linux-test) and a lot of messing about with docker rules.

After lots of attempts of encoding various dependencies depending on
age of images and checksum checks I finally settled on the simple
expedient of disabling the building of docker images when a sub-make
is invoked from the main makefile. We still check the dockerfile
exists but we no longer rely on the caching effect of re-running the
build stage.

There is a bit of a hack for forcing NOCACHE=1 for the debian-sid
image that prevents it being re-built too often but the chances are
anytime you want to run tests against the less well supported sid
compilers you'll end up re-building. I'm happy to drop the NOCACHE
stuff if people think it's a bit too messy. It's mainly to avoid
question about build failures as the:

  || { echo "Failed to build - see debian-sid.docker notes"; exit 1; }

in sid related dockerfiles tends to get lost in the noise.

I'm planning on sending the pull request for this next week so any
remaining review comments are most welcome. Patches currently needing
review are:

: patch 0012/tests tcg multiarch don t hard code paths ports f.patch needs 
review
: patch 0029/tests docker Makefile.include fix mipsel cross de.patch needs 
review
: patch 0048/tests add top level make dependency for docker bu.patch needs 
review
: patch 0049/docker docker.py don t conflate checksums for ext.patch needs 
review
: patch 0050/docker docker.py add check sub command.patch needs review
: patch 0051/tests Makefile call sub makes with SKIP_DOCKER_BU.patch needs 
review
: patch 0052/docker docker.py adding age check command.patch needs review
: patch 0053/tests docker Makefile.include only force SID to N.patch needs 
review

Cheers,

Alex Bennée (52):
  configure: add support for --cross-cc-FOO
  configure: move i386_cc to cross_cc_i386
  configure: allow user to specify --cross-cc-cflags-foo=
  configure: set cross_cc_FOO for host compiler
  docker: Add "cc" subcommand
  docker: extend "cc" command to accept compiler
  docker: allow "cc" command to run in user context
  docker: Makefile.include introduce DOCKER_SCRIPT
  tests/tcg: move architecture independent tests into subdir
  tests/tcg/multiarch: enable additional linux-test tests
  tests/tcg/multiarch: don't hard code paths/ports for linux-test
  tests/tcg/multiarch: move most output to stdout
  tests/tcg: move i386 specific tests into subdir
  tests/tcg: enable building for i386
  tests/tcg/i386: fix test-i386
  tests/tcg/i386: add runner for test-i386-fprem
  tests/tcg/x86_64: add Makefile.target
  tests/tcg/i386/test-i386: use modern vector_size attributes
  tests/tcg/i386/test-i386: fix printf format
  tests/tcg: move ARM specific tests into subdir
  tests/tcg: enable building for ARM
  tests/tcg/arm: fix up test-arm-iwmmxt test
  tests/tcg: enable building for AArch64
  tests/tcg/arm: add fcvt test cases for AArch32/64
  tests/tcg: move MIPS specific tests into subdir
  tests/tcg: enable building for MIPS
  tests/docker/Makefile.include: fix mipsel-cross dependancy
  tests/tcg/mips: include common mips hello-mips
  tests/tcg: enable building for s390x
  tests/tcg: enable building for ppc64
  tests/tcg: enable building for Alpha
  tests/tcg/alpha: add Alpha specific tests
  tests/tcg: enable building for HPPA
  tests/tcg: enable building for m68k
  tests/tcg: enable building for sh4
  tests/tcg: enable building for sparc64
  tests/tcg: enable building for mips64
  tests/tcg: enable building for RISCV64
  docker: move debian-powerpc-cross to sid based build
  tests/tcg: enable building for PowerPC
  tests/tcg/Makefile: update to be called from Makefile.target
  Makefile.target: add (clean-/build-)guest-tests targets
  tests/Makefile.include: add [build|clean|check]-tcg targets
  tests/tcg: add run, diff, and skip helper macros
  tests/tcg: override runners for broken tests
  tests: add top-level make dependency for docker builds
  docker: docker.py don't conflate checksums for extra_files
  docker: docker.py add check sub-command
  tests/Makefile: call sub-makes with SKIP_DOCKER_BUILD=1
  docker: docker.py adding age check command
  tests/docker/Makefile.include: only force SID to NOCACHE if old
  .travis.yml: add check-tcg test

Fam Zheng (2):
  tests/tcg/multiarch: Build fix for linux-test
  tests/tcg/i386: Build fix for hello-i386

 .travis.yml   |8 +
 MAINTAINERS   |   14 +
 Makefile.target   |5 +
 configure |  134 +-
 tests/Makefile.include|   41 +-
 tests/docker/Makefile.include |   47 +-
 tests/docker/docker.py|   91 +-
 .../dockerfiles/debian-alpha-cross.docker |   12 +
 

[Qemu-devel] [PULL v2 08/19] translate-all: remove hole in PageDesc

[Richard Henderson]

From: "Emilio G. Cota" 

Groundwork for supporting parallel TCG generation.

Move the hole to the end of the struct, so that a u32
field can be added there without bloating the struct.

Reviewed-by: Richard Henderson 
Reviewed-by: Alex Bennée 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 accel/tcg/translate-all.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 52273e5846..b9c36a3e45 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -107,8 +107,8 @@ typedef struct PageDesc {
 #ifdef CONFIG_SOFTMMU
 /* in order to optimize self modifying code, we count the number
of lookups we do to a given page to use a bitmap */
-unsigned int code_write_count;
 unsigned long *code_bitmap;
+unsigned int code_write_count;
 #else
 unsigned long flags;
 #endif
-- 
2.17.1

[Qemu-devel] [PULL v2 15/19] translate-all: protect TB jumps with a per-destination-TB lock

[Richard Henderson]

From: "Emilio G. Cota" 

This applies to both user-mode and !user-mode emulation.

Instead of relying on a global lock, protect the list of incoming
jumps with tb->jmp_lock. This lock also protects tb->cflags,
so update all tb->cflags readers outside tb->jmp_lock to use
atomic reads via tb_cflags().

In order to find the destination TB (and therefore its jmp_lock)
from the origin TB, we introduce tb->jmp_dest[].

I considered not using a linked list of jumps, which simplifies
code and makes the struct smaller. However, it unnecessarily increases
memory usage, which results in a performance decrease. See for
instance these numbers booting+shutting down debian-arm:
  Time (s)  Rel. err (%)  Abs. err (s)  Rel. slowdown (%)
--
 before  20.88  0.74  0.154512 0.
 after   20.81  0.38  0.079078-0.33524904
 GTree   21.02  0.28  0.058856 0.67049808
 GHashTable + xxhash 21.63  1.08  0.233604  3.5919540

Using a hash table or a binary tree to keep track of the jumps
doesn't really pay off, not only due to the increased memory usage,
but also because most TBs have only 0 or 1 jumps to them. The maximum
number of jumps when booting debian-arm that I measured is 35, but
as we can see in the histogram below a TB with that many incoming jumps
is extremely rare; the average TB has 0.80 incoming jumps.

n_jumps: 379208; avg jumps/tb: 0.801099
dist: [0.0,1.0)|▄█▁▁▁ ▁▁ ▁▁▁  ▁▁▁ ▁|[34.0,35.0]

Reviewed-by: Richard Henderson 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 include/exec/exec-all.h |  35 ++
 accel/tcg/cpu-exec.c|  41 +++
 accel/tcg/translate-all.c   | 120 +++-
 docs/devel/multi-thread-tcg.txt |   6 +-
 4 files changed, 125 insertions(+), 77 deletions(-)

diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 4f07a17052..3c2a0efb55 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -345,7 +345,7 @@ struct TranslationBlock {
 #define CF_LAST_IO 0x8000 /* Last insn may be an IO access.  */
 #define CF_NOCACHE 0x0001 /* To be freed after execution */
 #define CF_USE_ICOUNT  0x0002
-#define CF_INVALID 0x0004 /* TB is stale. Setters need tb_lock */
+#define CF_INVALID 0x0004 /* TB is stale. Set with @jmp_lock held */
 #define CF_PARALLEL0x0008 /* Generate code for a parallel context */
 /* cflags' mask for hashing/comparison */
 #define CF_HASH_MASK   \
@@ -364,6 +364,9 @@ struct TranslationBlock {
 uintptr_t page_next[2];
 tb_page_addr_t page_addr[2];
 
+/* jmp_lock placed here to fill a 4-byte hole. Its documentation is below 
*/
+QemuSpin jmp_lock;
+
 /* The following data are used to directly call another TB from
  * the code of this one. This can be done either by emitting direct or
  * indirect native jump instructions. These jumps are reset so that the TB
@@ -375,20 +378,26 @@ struct TranslationBlock {
 #define TB_JMP_RESET_OFFSET_INVALID 0x /* indicates no jump generated */
 uintptr_t jmp_target_arg[2];  /* target address or offset */
 
-/* Each TB has an associated circular list of TBs jumping to this one.
- * jmp_list_first points to the first TB jumping to this one.
- * jmp_list_next is used to point to the next TB in a list.
- * Since each TB can have two jumps, it can participate in two lists.
- * jmp_list_first and jmp_list_next are 4-byte aligned pointers to a
- * TranslationBlock structure, but the two least significant bits of
- * them are used to encode which data field of the pointed TB should
- * be used to traverse the list further from that TB:
- * 0 => jmp_list_next[0], 1 => jmp_list_next[1], 2 => jmp_list_first.
- * In other words, 0/1 tells which jump is used in the pointed TB,
- * and 2 means that this is a pointer back to the target TB of this list.
+/*
+ * Each TB has a NULL-terminated list (jmp_list_head) of incoming jumps.
+ * Each TB can have two outgoing jumps, and therefore can participate
+ * in two lists. The list entries are kept in jmp_list_next[2]. The least
+ * significant bit (LSB) of the pointers in these lists is used to encode
+ * which of the two list entries is to be used in the pointed TB.
+ *
+ * List traversals are protected by jmp_lock. The destination TB of each
+ * outgoing jump is kept in jmp_dest[] so that the appropriate jmp_lock
+ * can be acquired from any origin TB.
+ *
+ * jmp_dest[] are tagged pointers as well. The LSB is set when the TB is
+ * being invalidated, so that no further outgoing jumps from it can be set.
+ *
+ * jmp_lock also protects the CF_INVALID cflag; a jump must not be chained
+ * to a 

[Qemu-devel] [PULL v2 06/19] translate-all: iterate over TBs in a page with PAGE_FOR_EACH_TB

[Richard Henderson]

From: "Emilio G. Cota" 

This commit does several things, but to avoid churn I merged them all
into the same commit. To wit:

- Use uintptr_t instead of TranslationBlock * for the list of TBs in a page.
  Just like we did in (c37e6d7e "tcg: Use uintptr_t type for
  jmp_list_{next|first} fields of TB"), the rationale is the same: these
  are tagged pointers, not pointers. So use a more appropriate type.

- Only check the least significant bit of the tagged pointers. Masking
  with 3/~3 is unnecessary and confusing.

- Introduce the TB_FOR_EACH_TAGGED macro, and use it to define
  PAGE_FOR_EACH_TB, which improves readability. Note that
  TB_FOR_EACH_TAGGED will gain another user in a subsequent patch.

- Update tb_page_remove to use PAGE_FOR_EACH_TB. In case there
  is a bug and we attempt to remove a TB that is not in the list, instead
  of segfaulting (since the list is NULL-terminated) we will reach
  g_assert_not_reached().

Reviewed-by: Richard Henderson 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 include/exec/exec-all.h   |  2 +-
 accel/tcg/translate-all.c | 62 ++-
 2 files changed, 30 insertions(+), 34 deletions(-)

diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 8d4306ac25..07653d3c92 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -360,7 +360,7 @@ struct TranslationBlock {
 struct TranslationBlock *orig_tb;
 /* first and second physical page containing code. The lower bit
of the pointer tells the index in page_next[] */
-struct TranslationBlock *page_next[2];
+uintptr_t page_next[2];
 tb_page_addr_t page_addr[2];
 
 /* The following data are used to directly call another TB from
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index a9f2bfb468..52e62125ed 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -103,7 +103,7 @@
 
 typedef struct PageDesc {
 /* list of TBs intersecting this ram page */
-TranslationBlock *first_tb;
+uintptr_t first_tb;
 #ifdef CONFIG_SOFTMMU
 /* in order to optimize self modifying code, we count the number
of lookups we do to a given page to use a bitmap */
@@ -114,6 +114,15 @@ typedef struct PageDesc {
 #endif
 } PageDesc;
 
+/* list iterators for lists of tagged pointers in TranslationBlock */
+#define TB_FOR_EACH_TAGGED(head, tb, n, field)  \
+for (n = (head) & 1, tb = (TranslationBlock *)((head) & ~1);\
+ tb; tb = (TranslationBlock *)tb->field[n], n = (uintptr_t)tb & 1, \
+ tb = (TranslationBlock *)((uintptr_t)tb & ~1))
+
+#define PAGE_FOR_EACH_TB(pagedesc, tb, n)   \
+TB_FOR_EACH_TAGGED((pagedesc)->first_tb, tb, n, page_next)
+
 /* In system mode we want L1_MAP to be based on ram offsets,
while in user mode we want it to be based on virtual addresses.  */
 #if !defined(CONFIG_USER_ONLY)
@@ -815,7 +824,7 @@ static void page_flush_tb_1(int level, void **lp)
 PageDesc *pd = *lp;
 
 for (i = 0; i < V_L2_SIZE; ++i) {
-pd[i].first_tb = NULL;
+pd[i].first_tb = (uintptr_t)NULL;
 invalidate_page_bitmap(pd + i);
 }
 } else {
@@ -943,21 +952,21 @@ static void tb_page_check(void)
 
 #endif /* CONFIG_USER_ONLY */
 
-static inline void tb_page_remove(TranslationBlock **ptb, TranslationBlock *tb)
+static inline void tb_page_remove(PageDesc *pd, TranslationBlock *tb)
 {
 TranslationBlock *tb1;
+uintptr_t *pprev;
 unsigned int n1;
 
-for (;;) {
-tb1 = *ptb;
-n1 = (uintptr_t)tb1 & 3;
-tb1 = (TranslationBlock *)((uintptr_t)tb1 & ~3);
+pprev = >first_tb;
+PAGE_FOR_EACH_TB(pd, tb1, n1) {
 if (tb1 == tb) {
-*ptb = tb1->page_next[n1];
-break;
+*pprev = tb1->page_next[n1];
+return;
 }
-ptb = >page_next[n1];
+pprev = >page_next[n1];
 }
+g_assert_not_reached();
 }
 
 /* remove the TB from a list of TBs jumping to the n-th jump target of the TB 
*/
@@ -1045,12 +1054,12 @@ void tb_phys_invalidate(TranslationBlock *tb, 
tb_page_addr_t page_addr)
 /* remove the TB from the page list */
 if (tb->page_addr[0] != page_addr) {
 p = page_find(tb->page_addr[0] >> TARGET_PAGE_BITS);
-tb_page_remove(>first_tb, tb);
+tb_page_remove(p, tb);
 invalidate_page_bitmap(p);
 }
 if (tb->page_addr[1] != -1 && tb->page_addr[1] != page_addr) {
 p = page_find(tb->page_addr[1] >> TARGET_PAGE_BITS);
-tb_page_remove(>first_tb, tb);
+tb_page_remove(p, tb);
 invalidate_page_bitmap(p);
 }
 
@@ -1081,10 +1090,7 @@ static void build_page_bitmap(PageDesc *p)
 
 p->code_bitmap = bitmap_new(TARGET_PAGE_SIZE);
 
-tb = p->first_tb;
-while (tb != NULL) {
-n = (uintptr_t)tb & 3;
-tb = (TranslationBlock *)((uintptr_t)tb & ~3);
+

[Qemu-devel] [PULL v2 13/19] translate-all: introduce assert_no_pages_locked

[Richard Henderson]

From: "Emilio G. Cota" 

The appended adds assertions to make sure we do not longjmp with page
locks held. Note that user-mode has nothing to check, since page_locks
are !user-mode only.

Reviewed-by: Richard Henderson 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 include/exec/exec-all.h   | 8 
 accel/tcg/cpu-exec.c  | 1 +
 accel/tcg/translate-all.c | 7 +++
 3 files changed, 16 insertions(+)

diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 8d92e3cea9..4f07a17052 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -435,6 +435,14 @@ void tb_lock(void);
 void tb_unlock(void);
 void tb_lock_reset(void);
 
+#if !defined(CONFIG_USER_ONLY) && defined(CONFIG_DEBUG_TCG)
+void assert_no_pages_locked(void);
+#else
+static inline void assert_no_pages_locked(void)
+{
+}
+#endif
+
 #if !defined(CONFIG_USER_ONLY)
 
 /**
diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
index 7570c59f09..d75c35380a 100644
--- a/accel/tcg/cpu-exec.c
+++ b/accel/tcg/cpu-exec.c
@@ -273,6 +273,7 @@ void cpu_exec_step_atomic(CPUState *cpu)
 tcg_debug_assert(!have_mmap_lock());
 #endif
 tb_lock_reset();
+assert_no_pages_locked();
 }
 
 if (in_exclusive_region) {
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 8b378586f4..c75298d08a 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -658,6 +658,12 @@ do_assert_page_locked(const PageDesc *pd, const char 
*file, int line)
 
 #define assert_page_locked(pd) do_assert_page_locked(pd, __FILE__, __LINE__)
 
+void assert_no_pages_locked(void)
+{
+ht_pages_locked_debug_init();
+g_assert(g_hash_table_size(ht_pages_locked_debug) == 0);
+}
+
 #else /* !CONFIG_DEBUG_TCG */
 
 #define assert_page_locked(pd)
@@ -829,6 +835,7 @@ page_collection_lock(tb_page_addr_t start, tb_page_addr_t 
end)
 set->tree = g_tree_new_full(tb_page_addr_cmp, NULL, NULL,
 page_entry_destroy);
 set->max = NULL;
+assert_no_pages_locked();
 
  retry:
 g_tree_foreach(set->tree, page_entry_lock, NULL);
-- 
2.17.1

[Qemu-devel] [PULL v2 17/19] translate-all: remove tb_lock mention from cpu_restore_state_from_tb

[Richard Henderson]

From: "Emilio G. Cota" 

tb_lock was needed when the function did retranslation. However,
since fca8a500d519 ("tcg: Save insn data and use it in
cpu_restore_state_from_tb") we don't do retranslation.

Get rid of the comment.

Reviewed-by: Richard Henderson 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 accel/tcg/translate-all.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 55c9e1b196..b33cd9bc98 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -355,7 +355,6 @@ static int encode_search(TranslationBlock *tb, uint8_t 
*block)
 }
 
 /* The cpu state corresponding to 'searched_pc' is restored.
- * Called with tb_lock held.
  * When reset_icount is true, current TB will be interrupted and
  * icount should be recalculated.
  */
-- 
2.17.1

[Qemu-devel] [PULL v2 14/19] translate-all: discard TB when tb_link_page returns an existing matching TB

[Richard Henderson]

From: "Emilio G. Cota" 

Use the recently-gained QHT feature of returning the matching TB if it
already exists. This allows us to get rid of the lookup we perform
right after acquiring tb_lock.

Suggested-by: Richard Henderson 
Reviewed-by: Richard Henderson 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 accel/tcg/cpu-exec.c| 14 ++---
 accel/tcg/translate-all.c   | 50 +++--
 docs/devel/multi-thread-tcg.txt |  3 ++
 3 files changed, 46 insertions(+), 21 deletions(-)

diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
index d75c35380a..45f6ebc65e 100644
--- a/accel/tcg/cpu-exec.c
+++ b/accel/tcg/cpu-exec.c
@@ -245,10 +245,7 @@ void cpu_exec_step_atomic(CPUState *cpu)
 if (tb == NULL) {
 mmap_lock();
 tb_lock();
-tb = tb_htable_lookup(cpu, pc, cs_base, flags, cf_mask);
-if (likely(tb == NULL)) {
-tb = tb_gen_code(cpu, pc, cs_base, flags, cflags);
-}
+tb = tb_gen_code(cpu, pc, cs_base, flags, cflags);
 tb_unlock();
 mmap_unlock();
 }
@@ -398,14 +395,7 @@ static inline TranslationBlock *tb_find(CPUState *cpu,
 tb_lock();
 acquired_tb_lock = true;
 
-/* There's a chance that our desired tb has been translated while
- * taking the locks so we check again inside the lock.
- */
-tb = tb_htable_lookup(cpu, pc, cs_base, flags, cf_mask);
-if (likely(tb == NULL)) {
-/* if no translated code available, then translate it now */
-tb = tb_gen_code(cpu, pc, cs_base, flags, cf_mask);
-}
+tb = tb_gen_code(cpu, pc, cs_base, flags, cf_mask);
 
 mmap_unlock();
 /* We add the TB in the virtual pc hash table for the fast lookup */
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index c75298d08a..3f977532bf 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -1581,18 +1581,30 @@ static inline void tb_page_add(PageDesc *p, 
TranslationBlock *tb,
  * (-1) to indicate that only one page contains the TB.
  *
  * Called with mmap_lock held for user-mode emulation.
+ *
+ * Returns a pointer @tb, or a pointer to an existing TB that matches @tb.
+ * Note that in !user-mode, another thread might have already added a TB
+ * for the same block of guest code that @tb corresponds to. In that case,
+ * the caller should discard the original @tb, and use instead the returned TB.
  */
-static void tb_link_page(TranslationBlock *tb, tb_page_addr_t phys_pc,
- tb_page_addr_t phys_page2)
+static TranslationBlock *
+tb_link_page(TranslationBlock *tb, tb_page_addr_t phys_pc,
+ tb_page_addr_t phys_page2)
 {
 PageDesc *p;
 PageDesc *p2 = NULL;
+void *existing_tb = NULL;
 uint32_t h;
 
 assert_memory_lock();
 
 /*
  * Add the TB to the page list, acquiring first the pages's locks.
+ * We keep the locks held until after inserting the TB in the hash table,
+ * so that if the insertion fails we know for sure that the TBs are still
+ * in the page descriptors.
+ * Note that inserting into the hash table first isn't an option, since
+ * we can only insert TBs that are fully initialized.
  */
 page_lock_pair(, phys_pc, , phys_page2, 1);
 tb_page_add(p, tb, 0, phys_pc & TARGET_PAGE_MASK);
@@ -1602,21 +1614,33 @@ static void tb_link_page(TranslationBlock *tb, 
tb_page_addr_t phys_pc,
 tb->page_addr[1] = -1;
 }
 
+/* add in the hash table */
+h = tb_hash_func(phys_pc, tb->pc, tb->flags, tb->cflags & CF_HASH_MASK,
+ tb->trace_vcpu_dstate);
+qht_insert(_ctx.htable, tb, h, _tb);
+
+/* remove TB from the page(s) if we couldn't insert it */
+if (unlikely(existing_tb)) {
+tb_page_remove(p, tb);
+invalidate_page_bitmap(p);
+if (p2) {
+tb_page_remove(p2, tb);
+invalidate_page_bitmap(p2);
+}
+tb = existing_tb;
+}
+
 if (p2) {
 page_unlock(p2);
 }
 page_unlock(p);
 
-/* add in the hash table */
-h = tb_hash_func(phys_pc, tb->pc, tb->flags, tb->cflags & CF_HASH_MASK,
- tb->trace_vcpu_dstate);
-qht_insert(_ctx.htable, tb, h, NULL);
-
 #ifdef CONFIG_USER_ONLY
 if (DEBUG_TB_CHECK_GATE) {
 tb_page_check();
 }
 #endif
+return tb;
 }
 
 /* Called with mmap_lock held for user mode emulation.  */
@@ -1625,7 +1649,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
   uint32_t flags, int cflags)
 {
 CPUArchState *env = cpu->env_ptr;
-TranslationBlock *tb;
+TranslationBlock *tb, *existing_tb;
 tb_page_addr_t phys_pc, phys_page2;
 target_ulong virt_page2;
 tcg_insn_unit *gen_code_buf;
@@ -1773,7 +1797,15 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
  * memory barrier is required before 

[Qemu-devel] [PULL v2 05/19] tcg: move tb_ctx.tb_phys_invalidate_count to tcg_ctx

[Richard Henderson]

From: "Emilio G. Cota" 

Thereby making it per-TCGContext. Once we remove tb_lock, this will
avoid an atomic increment every time a TB is invalidated.

Reviewed-by: Richard Henderson 
Reviewed-by: Alex Bennée 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 include/exec/tb-context.h |  1 -
 tcg/tcg.h |  3 +++
 accel/tcg/translate-all.c |  5 +++--
 tcg/tcg.c | 14 ++
 4 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/include/exec/tb-context.h b/include/exec/tb-context.h
index d8472c88fb..8c9b49c98e 100644
--- a/include/exec/tb-context.h
+++ b/include/exec/tb-context.h
@@ -37,7 +37,6 @@ struct TBContext {
 
 /* statistics */
 unsigned tb_flush_count;
-int tb_phys_invalidate_count;
 };
 
 extern TBContext tb_ctx;
diff --git a/tcg/tcg.h b/tcg/tcg.h
index 1e6df1906f..e49b289ba1 100644
--- a/tcg/tcg.h
+++ b/tcg/tcg.h
@@ -695,6 +695,8 @@ struct TCGContext {
 /* Threshold to flush the translated code buffer.  */
 void *code_gen_highwater;
 
+size_t tb_phys_invalidate_count;
+
 /* Track which vCPU triggers events */
 CPUState *cpu;  /* *_trans */
 
@@ -868,6 +870,7 @@ size_t tcg_code_capacity(void);
 
 void tcg_tb_insert(TranslationBlock *tb);
 void tcg_tb_remove(TranslationBlock *tb);
+size_t tcg_tb_phys_invalidate_count(void);
 TranslationBlock *tcg_tb_lookup(uintptr_t tc_ptr);
 void tcg_tb_foreach(GTraverseFunc func, gpointer user_data);
 size_t tcg_nb_tbs(void);
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index ef841c82cc..a9f2bfb468 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -1069,7 +1069,8 @@ void tb_phys_invalidate(TranslationBlock *tb, 
tb_page_addr_t page_addr)
 /* suppress any remaining jumps to this TB */
 tb_jmp_unlink(tb);
 
-tb_ctx.tb_phys_invalidate_count++;
+atomic_set(_ctx->tb_phys_invalidate_count,
+   tcg_ctx->tb_phys_invalidate_count + 1);
 }
 
 #ifdef CONFIG_SOFTMMU
@@ -1855,7 +1856,7 @@ void dump_exec_info(FILE *f, fprintf_function cpu_fprintf)
 cpu_fprintf(f, "\nStatistics:\n");
 cpu_fprintf(f, "TB flush count  %u\n",
 atomic_read(_ctx.tb_flush_count));
-cpu_fprintf(f, "TB invalidate count %d\n", 
tb_ctx.tb_phys_invalidate_count);
+cpu_fprintf(f, "TB invalidate count %zu\n", 
tcg_tb_phys_invalidate_count());
 cpu_fprintf(f, "TLB flush count %zu\n", tlb_flush_count());
 tcg_dump_info(f, cpu_fprintf);
 }
diff --git a/tcg/tcg.c b/tcg/tcg.c
index 62e3391020..1d1dfd7f7c 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -791,6 +791,20 @@ size_t tcg_code_capacity(void)
 return capacity;
 }
 
+size_t tcg_tb_phys_invalidate_count(void)
+{
+unsigned int n_ctxs = atomic_read(_tcg_ctxs);
+unsigned int i;
+size_t total = 0;
+
+for (i = 0; i < n_ctxs; i++) {
+const TCGContext *s = atomic_read(_ctxs[i]);
+
+total += atomic_read(>tb_phys_invalidate_count);
+}
+return total;
+}
+
 /* pool based memory allocation */
 void *tcg_malloc_internal(TCGContext *s, int size)
 {
-- 
2.17.1

[Qemu-devel] [PULL v2 04/19] tcg: track TBs with per-region BST's

[Richard Henderson]

From: "Emilio G. Cota" 

This paves the way for enabling scalable parallel generation of TCG code.

Instead of tracking TBs with a single binary search tree (BST), use a
BST for each TCG region, protecting it with a lock. This is as scalable
as it gets, since each TCG thread operates on a separate region.

The core of this change is the introduction of struct tcg_region_tree,
which contains a pointer to a GTree and an associated lock to serialize
accesses to it. We then allocate an array of tcg_region_tree's, adding
the appropriate padding to avoid false sharing based on
qemu_dcache_linesize.

Given a tc_ptr, we first find the corresponding region_tree. This
is done by special-casing the first and last regions first, since they
might be of size != region.size; otherwise we just divide the offset
by region.stride. I was worried about this division (several dozen
cycles of latency), but profiling shows that this is not a fast path.
Note that region.stride is not required to be a power of two; it
is only required to be a multiple of the host's page size.

Note that with this design we can also provide consistent snapshots
about all region trees at once; for instance, tcg_tb_foreach
acquires/releases all region_tree locks before/after iterating over them.
For this reason we now drop tb_lock in dump_exec_info().

As an alternative I considered implementing a concurrent BST, but this
can be tricky to get right, offers no consistent snapshots of the BST,
and performance and scalability-wise I don't think it could ever beat
having separate GTrees, given that our workload is insert-mostly (all
concurrent BST designs I've seen focus, understandably, on making
lookups fast, which comes at the expense of convoluted, non-wait-free
insertions/removals).

Reviewed-by: Richard Henderson 
Reviewed-by: Alex Bennée 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 include/exec/exec-all.h   |   1 -
 include/exec/tb-context.h |   1 -
 tcg/tcg.h |   6 ++
 accel/tcg/cpu-exec.c  |   2 +-
 accel/tcg/translate-all.c | 101 +++-
 tcg/tcg.c | 191 ++
 6 files changed, 213 insertions(+), 89 deletions(-)

diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 8bbea787a9..8d4306ac25 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -405,7 +405,6 @@ static inline uint32_t curr_cflags(void)
  | (use_icount ? CF_USE_ICOUNT : 0);
 }
 
-void tb_remove(TranslationBlock *tb);
 void tb_flush(CPUState *cpu);
 void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
 TranslationBlock *tb_htable_lookup(CPUState *cpu, target_ulong pc,
diff --git a/include/exec/tb-context.h b/include/exec/tb-context.h
index 1d41202485..d8472c88fb 100644
--- a/include/exec/tb-context.h
+++ b/include/exec/tb-context.h
@@ -31,7 +31,6 @@ typedef struct TBContext TBContext;
 
 struct TBContext {
 
-GTree *tb_tree;
 struct qht htable;
 /* any access to the tbs or the page table must use this lock */
 QemuMutex tb_lock;
diff --git a/tcg/tcg.h b/tcg/tcg.h
index 509f4d65d2..1e6df1906f 100644
--- a/tcg/tcg.h
+++ b/tcg/tcg.h
@@ -866,6 +866,12 @@ void tcg_region_reset_all(void);
 size_t tcg_code_size(void);
 size_t tcg_code_capacity(void);
 
+void tcg_tb_insert(TranslationBlock *tb);
+void tcg_tb_remove(TranslationBlock *tb);
+TranslationBlock *tcg_tb_lookup(uintptr_t tc_ptr);
+void tcg_tb_foreach(GTraverseFunc func, gpointer user_data);
+size_t tcg_nb_tbs(void);
+
 /* user-mode: Called with tb_lock held.  */
 static inline void *tcg_malloc(int size)
 {
diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
index 6d6c51b686..7570c59f09 100644
--- a/accel/tcg/cpu-exec.c
+++ b/accel/tcg/cpu-exec.c
@@ -224,7 +224,7 @@ static void cpu_exec_nocache(CPUState *cpu, int max_cycles,
 
 tb_lock();
 tb_phys_invalidate(tb, -1);
-tb_remove(tb);
+tcg_tb_remove(tb);
 tb_unlock();
 }
 #endif
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 1695f8c352..ef841c82cc 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -205,8 +205,6 @@ void tb_lock_reset(void)
 }
 }
 
-static TranslationBlock *tb_find_pc(uintptr_t tc_ptr);
-
 void cpu_gen_init(void)
 {
 tcg_context_init(_init_ctx);
@@ -375,13 +373,13 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t host_pc, 
bool will_exit)
 
 if (check_offset < tcg_init_ctx.code_gen_buffer_size) {
 tb_lock();
-tb = tb_find_pc(host_pc);
+tb = tcg_tb_lookup(host_pc);
 if (tb) {
 cpu_restore_state_from_tb(cpu, tb, host_pc, will_exit);
 if (tb->cflags & CF_NOCACHE) {
 /* one-shot translation, invalidate it immediately */
 tb_phys_invalidate(tb, -1);
-tb_remove(tb);
+tcg_tb_remove(tb);
 }
 r = true;
 }
@@ -728,48 +726,6 @@ static inline void 

[Qemu-devel] [PULL v2 03/19] qht: return existing entry when qht_insert fails

[Richard Henderson]

From: "Emilio G. Cota" 

The meaning of "existing" is now changed to "matches in hash and
ht->cmp result". This is saner than just checking the pointer value.

Suggested-by: Richard Henderson 
Reviewed-by:  Richard Henderson 
Reviewed-by: Alex Bennée 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 include/qemu/qht.h|  7 +--
 accel/tcg/translate-all.c |  2 +-
 tests/qht-bench.c |  4 ++--
 tests/test-qht.c  |  8 +++-
 util/qht.c| 27 +--
 5 files changed, 32 insertions(+), 16 deletions(-)

diff --git a/include/qemu/qht.h b/include/qemu/qht.h
index 5f03a0f4cb..1fb9116fa0 100644
--- a/include/qemu/qht.h
+++ b/include/qemu/qht.h
@@ -70,6 +70,7 @@ void qht_destroy(struct qht *ht);
  * @ht: QHT to insert to
  * @p: pointer to be inserted
  * @hash: hash corresponding to @p
+ * @existing: address where the pointer to an existing entry can be copied to
  *
  * Attempting to insert a NULL @p is a bug.
  * Inserting the same pointer @p with different @hash values is a bug.
@@ -78,9 +79,11 @@ void qht_destroy(struct qht *ht);
  * inserted into the hash table.
  *
  * Returns true on success.
- * Returns false if the @p-@hash pair already exists in the hash table.
+ * Returns false if there is an existing entry in the table that is equivalent
+ * (i.e. ht->cmp matches and the hash is the same) to @p-@h. If @existing
+ * is !NULL, a pointer to this existing entry is copied to it.
  */
-bool qht_insert(struct qht *ht, void *p, uint32_t hash);
+bool qht_insert(struct qht *ht, void *p, uint32_t hash, void **existing);
 
 /**
  * qht_lookup_custom - Look up a pointer using a custom comparison function.
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index f39123bd5a..1695f8c352 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -1242,7 +1242,7 @@ static void tb_link_page(TranslationBlock *tb, 
tb_page_addr_t phys_pc,
 /* add in the hash table */
 h = tb_hash_func(phys_pc, tb->pc, tb->flags, tb->cflags & CF_HASH_MASK,
  tb->trace_vcpu_dstate);
-qht_insert(_ctx.htable, tb, h);
+qht_insert(_ctx.htable, tb, h, NULL);
 
 #ifdef CONFIG_USER_ONLY
 if (DEBUG_TB_CHECK_GATE) {
diff --git a/tests/qht-bench.c b/tests/qht-bench.c
index c94ac25158..f492b3a20a 100644
--- a/tests/qht-bench.c
+++ b/tests/qht-bench.c
@@ -163,7 +163,7 @@ static void do_rw(struct thread_info *info)
 bool written = false;
 
 if (qht_lookup(, p, hash) == NULL) {
-written = qht_insert(, p, hash);
+written = qht_insert(, p, hash, NULL);
 }
 if (written) {
 stats->in++;
@@ -322,7 +322,7 @@ static void htable_init(void)
 r = xorshift64star(r);
 p = [r & (init_range - 1)];
 hash = h(*p);
-if (qht_insert(, p, hash)) {
+if (qht_insert(, p, hash, NULL)) {
 break;
 }
 retries++;
diff --git a/tests/test-qht.c b/tests/test-qht.c
index b069881342..dda6a067be 100644
--- a/tests/test-qht.c
+++ b/tests/test-qht.c
@@ -27,11 +27,17 @@ static void insert(int a, int b)
 
 for (i = a; i < b; i++) {
 uint32_t hash;
+void *existing;
+bool inserted;
 
 arr[i] = i;
 hash = i;
 
-qht_insert(, [i], hash);
+inserted = qht_insert(, [i], hash, NULL);
+g_assert_true(inserted);
+inserted = qht_insert(, [i], hash, );
+g_assert_false(inserted);
+g_assert_true(existing == [i]);
 }
 }
 
diff --git a/util/qht.c b/util/qht.c
index 487e9237bc..c138777a9c 100644
--- a/util/qht.c
+++ b/util/qht.c
@@ -511,9 +511,9 @@ void *qht_lookup(struct qht *ht, const void *userp, 
uint32_t hash)
 }
 
 /* call with head->lock held */
-static bool qht_insert__locked(struct qht *ht, struct qht_map *map,
-   struct qht_bucket *head, void *p, uint32_t hash,
-   bool *needs_resize)
+static void *qht_insert__locked(struct qht *ht, struct qht_map *map,
+struct qht_bucket *head, void *p, uint32_t 
hash,
+bool *needs_resize)
 {
 struct qht_bucket *b = head;
 struct qht_bucket *prev = NULL;
@@ -523,8 +523,9 @@ static bool qht_insert__locked(struct qht *ht, struct 
qht_map *map,
 do {
 for (i = 0; i < QHT_BUCKET_ENTRIES; i++) {
 if (b->pointers[i]) {
-if (unlikely(b->pointers[i] == p)) {
-return false;
+if (unlikely(b->hashes[i] == hash &&
+ ht->cmp(b->pointers[i], p))) {
+return b->pointers[i];
 }
 } else {
 goto found;
@@ -553,7 +554,7 @@ static bool qht_insert__locked(struct qht *ht, struct 
qht_map *map,
 atomic_set(>hashes[i], hash);
 atomic_set(>pointers[i], p);

[Qemu-devel] [PULL v2 07/19] translate-all: make l1_map lockless

[Richard Henderson]

From: "Emilio G. Cota" 

Groundwork for supporting parallel TCG generation.

We never remove entries from the radix tree, so we can use cmpxchg
to implement lockless insertions.

Reviewed-by: Richard Henderson 
Reviewed-by: Alex Bennée 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 accel/tcg/translate-all.c   | 24 ++--
 docs/devel/multi-thread-tcg.txt |  4 ++--
 2 files changed, 16 insertions(+), 12 deletions(-)

diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 52e62125ed..52273e5846 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -469,20 +469,12 @@ static void page_init(void)
 #endif
 }
 
-/* If alloc=1:
- * Called with tb_lock held for system emulation.
- * Called with mmap_lock held for user-mode emulation.
- */
 static PageDesc *page_find_alloc(tb_page_addr_t index, int alloc)
 {
 PageDesc *pd;
 void **lp;
 int i;
 
-if (alloc) {
-assert_memory_lock();
-}
-
 /* Level 1.  Always allocated.  */
 lp = l1_map + ((index >> v_l1_shift) & (v_l1_size - 1));
 
@@ -491,11 +483,17 @@ static PageDesc *page_find_alloc(tb_page_addr_t index, 
int alloc)
 void **p = atomic_rcu_read(lp);
 
 if (p == NULL) {
+void *existing;
+
 if (!alloc) {
 return NULL;
 }
 p = g_new0(void *, V_L2_SIZE);
-atomic_rcu_set(lp, p);
+existing = atomic_cmpxchg(lp, NULL, p);
+if (unlikely(existing)) {
+g_free(p);
+p = existing;
+}
 }
 
 lp = p + ((index >> (i * V_L2_BITS)) & (V_L2_SIZE - 1));
@@ -503,11 +501,17 @@ static PageDesc *page_find_alloc(tb_page_addr_t index, 
int alloc)
 
 pd = atomic_rcu_read(lp);
 if (pd == NULL) {
+void *existing;
+
 if (!alloc) {
 return NULL;
 }
 pd = g_new0(PageDesc, V_L2_SIZE);
-atomic_rcu_set(lp, pd);
+existing = atomic_cmpxchg(lp, NULL, pd);
+if (unlikely(existing)) {
+g_free(pd);
+pd = existing;
+}
 }
 
 return pd + (index & (V_L2_SIZE - 1));
diff --git a/docs/devel/multi-thread-tcg.txt b/docs/devel/multi-thread-tcg.txt
index a99b4564c6..faf8918b23 100644
--- a/docs/devel/multi-thread-tcg.txt
+++ b/docs/devel/multi-thread-tcg.txt
@@ -134,8 +134,8 @@ tb_set_jmp_target() code. Modification to the linked lists 
that allow
 searching for linked pages are done under the protect of the
 tb_lock().
 
-The global page table is protected by the tb_lock() in system-mode and
-mmap_lock() in linux-user mode.
+The global page table is a lockless radix tree; cmpxchg is used
+to atomically insert new elements.
 
 The lookup caches are updated atomically and the lookup hash uses QHT
 which is designed for concurrent safe lookup.
-- 
2.17.1

[Qemu-devel] [PULL v2 00/19] tcg queued patches

[Richard Henderson]

Since last:
  * fix clang build failure in patch 14,
  * include tb code gen overflow fix.


r~


Emilio G. Cota (17):
  qht: require a default comparison function
  qht: return existing entry when qht_insert fails
  tcg: track TBs with per-region BST's
  tcg: move tb_ctx.tb_phys_invalidate_count to tcg_ctx
  translate-all: iterate over TBs in a page with PAGE_FOR_EACH_TB
  translate-all: make l1_map lockless
  translate-all: remove hole in PageDesc
  translate-all: work page-by-page in tb_invalidate_phys_range_1
  translate-all: move tb_invalidate_phys_page_range up in the file
  translate-all: use per-page locking in !user-mode
  translate-all: add page_locked assertions
  translate-all: introduce assert_no_pages_locked
  translate-all: discard TB when tb_link_page returns an existing
matching TB
  translate-all: protect TB jumps with a per-destination-TB lock
  cputlb: remove tb_lock from tlb_flush functions
  translate-all: remove tb_lock mention from cpu_restore_state_from_tb
  tcg: remove tb_lock

John Arbuckle (1):
  tcg/i386: Use byte form of xgetbv instruction

Richard Henderson (1):
  tcg: Reduce max TB opcode count

 accel/tcg/translate-all.h   |6 +-
 include/exec/cpu-common.h   |2 +-
 include/exec/exec-all.h |   51 +-
 include/exec/memory-internal.h  |6 +-
 include/exec/tb-context.h   |4 -
 include/qemu/qht.h  |   32 +-
 tcg/tcg.h   |   19 +-
 accel/tcg/cpu-exec.c|   96 ++-
 accel/tcg/cputlb.c  |8 -
 accel/tcg/translate-all.c   | 1045 +--
 exec.c  |   26 +-
 linux-user/main.c   |3 -
 tcg/aarch64/tcg-target.inc.c|2 +-
 tcg/arm/tcg-target.inc.c|2 +-
 tcg/i386/tcg-target.inc.c   |7 +-
 tcg/mips/tcg-target.inc.c   |2 +-
 tcg/ppc/tcg-target.inc.c|4 +-
 tcg/s390/tcg-target.inc.c   |2 +-
 tcg/sparc/tcg-target.inc.c  |4 +-
 tcg/tcg.c   |  218 ++-
 tcg/tci/tcg-target.inc.c|2 +-
 tests/qht-bench.c   |   18 +-
 tests/test-qht.c|   23 +-
 util/qht.c  |   41 +-
 docs/devel/multi-thread-tcg.txt |   24 +-
 25 files changed, 1157 insertions(+), 490 deletions(-)

-- 
2.17.1

[Qemu-devel] [PULL v2 02/19] qht: require a default comparison function

[Richard Henderson]

From: "Emilio G. Cota" 

qht_lookup now uses the default cmp function. qht_lookup_custom is defined
to retain the old behaviour, that is a cmp function is explicitly provided.

qht_insert will gain use of the default cmp in the next patch.

Note that we move qht_lookup_custom's @func to be the last argument,
which makes the new qht_lookup as simple as possible.
Instead of this (i.e. keeping @func 2nd):
00010750 :
   10750:   89 d1   mov%edx,%ecx
   10752:   48 89 f2mov%rsi,%rdx
   10755:   48 8b 77 08 mov0x8(%rdi),%rsi
   10759:   e9 22 ff ff ff  jmpq   10680 
   1075e:   66 90   xchg   %ax,%ax

We get:
00010740 :
   10740:   48 8b 4f 08 mov0x8(%rdi),%rcx
   10744:   e9 37 ff ff ff  jmpq   10680 
   10749:   0f 1f 80 00 00 00 00nopl   0x0(%rax)

Reviewed-by: Richard Henderson 
Reviewed-by: Alex Bennée 
Signed-off-by: Emilio G. Cota 
Signed-off-by: Richard Henderson 
---
 include/qemu/qht.h| 25 -
 accel/tcg/cpu-exec.c  |  4 ++--
 accel/tcg/translate-all.c | 16 +++-
 tests/qht-bench.c | 14 +++---
 tests/test-qht.c  | 15 ++-
 util/qht.c| 14 +++---
 6 files changed, 65 insertions(+), 23 deletions(-)

diff --git a/include/qemu/qht.h b/include/qemu/qht.h
index 531aa95325..5f03a0f4cb 100644
--- a/include/qemu/qht.h
+++ b/include/qemu/qht.h
@@ -11,8 +11,11 @@
 #include "qemu/thread.h"
 #include "qemu/qdist.h"
 
+typedef bool (*qht_cmp_func_t)(const void *a, const void *b);
+
 struct qht {
 struct qht_map *map;
+qht_cmp_func_t cmp;
 QemuMutex lock; /* serializes setters of ht->map */
 unsigned int mode;
 };
@@ -47,10 +50,12 @@ typedef void (*qht_iter_func_t)(struct qht *ht, void *p, 
uint32_t h, void *up);
 /**
  * qht_init - Initialize a QHT
  * @ht: QHT to be initialized
+ * @cmp: default comparison function. Cannot be NULL.
  * @n_elems: number of entries the hash table should be optimized for.
  * @mode: bitmask with OR'ed QHT_MODE_*
  */
-void qht_init(struct qht *ht, size_t n_elems, unsigned int mode);
+void qht_init(struct qht *ht, qht_cmp_func_t cmp, size_t n_elems,
+  unsigned int mode);
 
 /**
  * qht_destroy - destroy a previously initialized QHT
@@ -78,11 +83,11 @@ void qht_destroy(struct qht *ht);
 bool qht_insert(struct qht *ht, void *p, uint32_t hash);
 
 /**
- * qht_lookup - Look up a pointer in a QHT
+ * qht_lookup_custom - Look up a pointer using a custom comparison function.
  * @ht: QHT to be looked up
- * @func: function to compare existing pointers against @userp
  * @userp: pointer to pass to @func
  * @hash: hash of the pointer to be looked up
+ * @func: function to compare existing pointers against @userp
  *
  * Needs to be called under an RCU read-critical section.
  *
@@ -94,8 +99,18 @@ bool qht_insert(struct qht *ht, void *p, uint32_t hash);
  * Returns the corresponding pointer when a match is found.
  * Returns NULL otherwise.
  */
-void *qht_lookup(struct qht *ht, qht_lookup_func_t func, const void *userp,
- uint32_t hash);
+void *qht_lookup_custom(struct qht *ht, const void *userp, uint32_t hash,
+qht_lookup_func_t func);
+
+/**
+ * qht_lookup - Look up a pointer in a QHT
+ * @ht: QHT to be looked up
+ * @userp: pointer to pass to the comparison function
+ * @hash: hash of the pointer to be looked up
+ *
+ * Calls qht_lookup_custom() using @ht's default comparison function.
+ */
+void *qht_lookup(struct qht *ht, const void *userp, uint32_t hash);
 
 /**
  * qht_remove - remove a pointer from the hash table
diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
index 4ef95d8dd3..6d6c51b686 100644
--- a/accel/tcg/cpu-exec.c
+++ b/accel/tcg/cpu-exec.c
@@ -295,7 +295,7 @@ struct tb_desc {
 uint32_t trace_vcpu_dstate;
 };
 
-static bool tb_cmp(const void *p, const void *d)
+static bool tb_lookup_cmp(const void *p, const void *d)
 {
 const TranslationBlock *tb = p;
 const struct tb_desc *desc = d;
@@ -340,7 +340,7 @@ TranslationBlock *tb_htable_lookup(CPUState *cpu, 
target_ulong pc,
 phys_pc = get_page_addr_code(desc.env, pc);
 desc.phys_page1 = phys_pc & TARGET_PAGE_MASK;
 h = tb_hash_func(phys_pc, pc, flags, cf_mask, *cpu->trace_dstate);
-return qht_lookup(_ctx.htable, tb_cmp, , h);
+return qht_lookup_custom(_ctx.htable, , h, tb_lookup_cmp);
 }
 
 void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr)
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index d48b56ca38..f39123bd5a 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -782,11 +782,25 @@ static inline void code_gen_alloc(size_t tb_size)
 qemu_mutex_init(_ctx.tb_lock);
 }
 
+static bool tb_cmp(const void *ap, const void *bp)
+{
+const TranslationBlock *a = ap;
+const TranslationBlock *b = bp;
+
+

[Qemu-devel] [PULL v2 01/19] tcg/i386: Use byte form of xgetbv instruction

[Richard Henderson]

From: John Arbuckle 

The assembler in most versions of Mac OS X is pretty old and does not
support the xgetbv instruction.  To go around this problem, the raw
encoding of the instruction is used instead.

Signed-off-by: John Arbuckle 
Message-Id: <20180604215102.11002-1-programmingk...@gmail.com>
Signed-off-by: Richard Henderson 
---
 tcg/i386/tcg-target.inc.c | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c
index 5357909fff..0d0ff524b7 100644
--- a/tcg/i386/tcg-target.inc.c
+++ b/tcg/i386/tcg-target.inc.c
@@ -3501,7 +3501,10 @@ static void tcg_target_init(TCGContext *s)
sure of not hitting invalid opcode.  */
 if (c & bit_OSXSAVE) {
 unsigned xcrl, xcrh;
-asm ("xgetbv" : "=a" (xcrl), "=d" (xcrh) : "c" (0));
+/* The xgetbv instruction is not available to older versions of
+ * the assembler, so we encode the instruction manually.
+ */
+asm(".byte 0x0f, 0x01, 0xd0" : "=a" (xcrl), "=d" (xcrh) : "c" (0));
 if ((xcrl & 6) == 6) {
 have_avx1 = (c & bit_AVX) != 0;
 have_avx2 = (b7 & bit_AVX2) != 0;
-- 
2.17.1

[Qemu-devel] [PULL 5/7] scripts/qemu.py: introduce set_console() method

[Eduardo Habkost]

From: Cleber Rosa 

The set_console() method is intended to ease higher level use cases
that require a console device.

The amount of intelligence is limited on purpose, requiring either the
device type explicitly, or the existence of a machine (pattern)
definition.

Because of the console device type selection criteria (by machine
type), users should also be able to define that.  It'll then be used
for both '-machine' and for the console device type selection.

Users of the set_console() method will certainly be interested in
accessing the console device, and for that a console_socket property
has been added.

Signed-off-by: Cleber Rosa 
Message-Id: <20180530184156.15634-5-cr...@redhat.com>
Tested-by: Philippe Mathieu-Daudé 
Reviewed-by: Stefan Hajnoczi 
Signed-off-by: Eduardo Habkost 
---
 scripts/qemu.py | 97 -
 1 file changed, 96 insertions(+), 1 deletion(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 7cd8193df8..f099ce7278 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -17,19 +17,41 @@ import logging
 import os
 import subprocess
 import qmp.qmp
+import re
 import shutil
+import socket
 import tempfile
 
 
 LOG = logging.getLogger(__name__)
 
 
+#: Maps machine types to the preferred console device types
+CONSOLE_DEV_TYPES = {
+r'^clipper$': 'isa-serial',
+r'^malta': 'isa-serial',
+r'^(pc.*|q35.*|isapc)$': 'isa-serial',
+r'^(40p|powernv|prep)$': 'isa-serial',
+r'^pseries.*': 'spapr-vty',
+r'^s390-ccw-virtio.*': 'sclpconsole',
+}
+
+
 class QEMUMachineError(Exception):
 """
 Exception called when an error in QEMUMachine happens.
 """
 
 
+class QEMUMachineAddDeviceError(QEMUMachineError):
+"""
+Exception raised when a request to add a device can not be fulfilled
+
+The failures are caused by limitations, lack of information or conflicting
+requests on the QEMUMachine methods.  This exception does not represent
+failures reported by the QEMU binary itself.
+"""
+
 class MonitorResponseError(qmp.qmp.QMPError):
 '''
 Represents erroneous QMP monitor reply
@@ -91,6 +113,10 @@ class QEMUMachine(object):
 self._test_dir = test_dir
 self._temp_dir = None
 self._launched = False
+self._machine = None
+self._console_device_type = None
+self._console_address = None
+self._console_socket = None
 
 # just in case logging wasn't configured by the main script:
 logging.basicConfig()
@@ -175,9 +201,19 @@ class QEMUMachine(object):
 self._monitor_address[1])
 else:
 moncdev = 'socket,id=mon,path=%s' % self._vm_monitor
-return ['-chardev', moncdev,
+args = ['-chardev', moncdev,
 '-mon', 'chardev=mon,mode=control',
 '-display', 'none', '-vga', 'none']
+if self._machine is not None:
+args.extend(['-machine', self._machine])
+if self._console_device_type is not None:
+self._console_address = os.path.join(self._temp_dir,
+ self._name + "-console.sock")
+chardev = ('socket,id=console,path=%s,server,nowait' %
+   self._console_address)
+device = '%s,chardev=console' % self._console_device_type
+args.extend(['-chardev', chardev, '-device', device])
+return args
 
 def _pre_launch(self):
 self._temp_dir = tempfile.mkdtemp(dir=self._test_dir)
@@ -202,6 +238,10 @@ class QEMUMachine(object):
 
 self._qemu_log_path = None
 
+if self._console_socket is not None:
+self._console_socket.close()
+self._console_socket = None
+
 if self._temp_dir is not None:
 shutil.rmtree(self._temp_dir)
 self._temp_dir = None
@@ -365,3 +405,58 @@ class QEMUMachine(object):
 Adds to the list of extra arguments to be given to the QEMU binary
 '''
 self._args.extend(args)
+
+def set_machine(self, machine_type):
+'''
+Sets the machine type
+
+If set, the machine type will be added to the base arguments
+of the resulting QEMU command line.
+'''
+self._machine = machine_type
+
+def set_console(self, device_type=None):
+'''
+Sets the device type for a console device
+
+If set, the console device and a backing character device will
+be added to the base arguments of the resulting QEMU command
+line.
+
+This is a convenience method that will either use the provided
+device type, of if not given, it will used the device type set
+on CONSOLE_DEV_TYPES.
+
+The actual setting of command line arguments will be be done at
+machine launch time, as it depends on the temporary directory
+to be created.
+
+@param device_type: the device type, such as "isa-serial"
+

[Qemu-devel] [PULL 6/7] Acceptance tests: add Linux kernel boot and console checking test

[Eduardo Habkost]

From: Cleber Rosa 

This test boots a Linux kernel, and checks that the given command
line was effective in two ways:

 * It makes the kernel use the set "console device" as a console
 * The kernel records the command line as expected in the console

Given that way too many error conditions may occur, and detecting the
kernel boot progress status may not be trivial, this test relies on a
timeout to handle unexpected situations.  Also, it's *not* tagged as a
quick test for obvious reasons.

It may be useful, while interactively running/debugging this test, or
tests similar to this one, to show some of the logging channels.
Example:

 $ avocado --show=QMP,console run boot_linux_console.py

Signed-off-by: Cleber Rosa 
Message-Id: <20180530184156.15634-6-cr...@redhat.com>
Reviewed-by: Stefan Hajnoczi 
Signed-off-by: Eduardo Habkost 
---
 tests/acceptance/boot_linux_console.py | 47 ++
 1 file changed, 47 insertions(+)
 create mode 100644 tests/acceptance/boot_linux_console.py

diff --git a/tests/acceptance/boot_linux_console.py 
b/tests/acceptance/boot_linux_console.py
new file mode 100644
index 00..98324f7591
--- /dev/null
+++ b/tests/acceptance/boot_linux_console.py
@@ -0,0 +1,47 @@
+# Functional test that boots a Linux kernel and checks the console
+#
+# Copyright (c) 2018 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa 
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import logging
+
+from avocado_qemu import Test
+
+
+class BootLinuxConsole(Test):
+"""
+Boots a x86_64 Linux kernel and checks that the console is operational
+and the kernel command line is properly passed from QEMU to the kernel
+
+:avocado: enable
+:avocado: tags=x86_64
+"""
+
+timeout = 60
+
+def test(self):
+kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
+  'Everything/x86_64/os/images/pxeboot/vmlinuz')
+kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
+kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+self.vm.set_machine('pc')
+self.vm.set_console()
+kernel_command_line = 'console=ttyS0'
+self.vm.add_args('-kernel', kernel_path,
+ '-append', kernel_command_line)
+self.vm.launch()
+console = self.vm.console_socket.makefile()
+console_logger = logging.getLogger('console')
+while True:
+msg = console.readline()
+console_logger.debug(msg.strip())
+if 'Kernel command line: %s' % kernel_command_line in msg:
+break
+if 'Kernel panic - not syncing' in msg:
+self.fail("Kernel panic reached")
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 2/7] Add functional/acceptance tests infrastructure

[Eduardo Habkost]

From: Cleber Rosa 

This patch adds the very minimum infrastructure necessary for writing
and running functional/acceptance tests, including:

 * Documentation
 * The avocado_qemu.Test base test class
 * One example tests (version.py)

Additional functionality is expected to be added along the tests that
require them.

Signed-off-by: Cleber Rosa 
Message-Id: <20180530184156.15634-2-cr...@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
[ehabkost: fix typo on testing.rst]
Reviewed-by: Stefan Hajnoczi 
Signed-off-by: Eduardo Habkost 
---
 docs/devel/testing.rst| 192 ++
 tests/acceptance/README.rst   |  10 ++
 tests/acceptance/avocado_qemu/__init__.py |  54 ++
 tests/acceptance/version.py   |  24 +++
 4 files changed, 280 insertions(+)
 create mode 100644 tests/acceptance/README.rst
 create mode 100644 tests/acceptance/avocado_qemu/__init__.py
 create mode 100644 tests/acceptance/version.py

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 0ca1a2d4b5..f33e5a8423 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -484,3 +484,195 @@ supported. To start the fuzzer, run
 
 Alternatively, some command different from "qemu-img info" can be tested, by
 changing the ``-c`` option.
+
+Acceptance tests using the Avocado Framework
+
+
+The ``tests/acceptance`` directory hosts functional tests, also known
+as acceptance level tests.  They're usually higher level tests, and
+may interact with external resources and with various guest operating
+systems.
+
+These tests are written using the Avocado Testing Framework (which must
+be installed separately) in conjunction with a the ``avocado_qemu.Test``
+class, implemented at ``tests/acceptance/avocado_qemu``.
+
+Tests based on ``avocado_qemu.Test`` can easily:
+
+ * Customize the command line arguments given to the convenience
+   ``self.vm`` attribute (a QEMUMachine instance)
+
+ * Interact with the QEMU monitor, send QMP commands and check
+   their results
+
+ * Interact with the guest OS, using the convenience console device
+   (which may be useful to assert the effectiveness and correctness of
+   command line arguments or QMP commands)
+
+ * Interact with external data files that accompany the test itself
+   (see ``self.get_data()``)
+
+ * Download (and cache) remote data files, such as firmware and kernel
+   images
+
+ * Have access to a library of guest OS images (by means of the
+   ``avocado.utils.vmimage`` library)
+
+ * Make use of various other test related utilities available at the
+   test class itself and at the utility library:
+
+   - 
http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
+   - 
http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
+
+Installation
+
+
+To install Avocado and its dependencies, run:
+
+.. code::
+
+  pip install --user avocado-framework
+
+Alternatively, follow the instructions on this link:
+
+  
http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#installing-avocado
+
+Overview
+
+
+This directory provides the ``avocado_qemu`` Python module, containing
+the ``avocado_qemu.Test`` class.  Here's a simple usage example:
+
+.. code::
+
+  from avocado_qemu import Test
+
+
+  class Version(Test):
+  """
+  :avocado: enable
+  :avocado: tags=quick
+  """
+  def test_qmp_human_info_version(self):
+  self.vm.launch()
+  res = self.vm.command('human-monitor-command',
+command_line='info version')
+  self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)')
+
+To execute your test, run:
+
+.. code::
+
+  avocado run version.py
+
+Tests may be classified according to a convention by using docstring
+directives such as ``:avocado: tags=TAG1,TAG2``.  To run all tests
+in the current directory, tagged as "quick", run:
+
+.. code::
+
+  avocado run -t quick .
+
+The ``avocado_qemu.Test`` base test class
+-
+
+The ``avocado_qemu.Test`` class has a number of characteristics that
+are worth being mentioned right away.
+
+First of all, it attempts to give each test a ready to use QEMUMachine
+instance, available at ``self.vm``.  Because many tests will tweak the
+QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
+is left to the test writer.
+
+At test "tear down", ``avocado_qemu.Test`` handles the QEMUMachine
+shutdown.
+
+QEMUMachine
+~~~
+
+The QEMUMachine API is already widely used in the Python iotests,
+device-crash-test and other Python scripts.  It's a wrapper around the
+execution of a QEMU binary, giving its users:
+
+ * the ability to set command line arguments to be given to the QEMU
+   binary
+
+ * a ready to use QMP connection and interface, which can be used to
+   send commands and inspect its results, as well 

[Qemu-devel] [PULL 4/7] Acceptance tests: add quick VNC tests

[Eduardo Habkost]

From: Cleber Rosa 

This patch adds a few simple behavior tests for VNC.

Signed-off-by: Cleber Rosa 
Reviewed-by: Stefan Hajnoczi 
Message-Id: <20180530184156.15634-4-cr...@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
Signed-off-by: Eduardo Habkost 
---
 tests/acceptance/vnc.py | 60 +
 1 file changed, 60 insertions(+)
 create mode 100644 tests/acceptance/vnc.py

diff --git a/tests/acceptance/vnc.py b/tests/acceptance/vnc.py
new file mode 100644
index 00..b1ef9d71b1
--- /dev/null
+++ b/tests/acceptance/vnc.py
@@ -0,0 +1,60 @@
+# Simple functional tests for VNC functionality
+#
+# Copyright (c) 2018 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa 
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+from avocado_qemu import Test
+
+
+class Vnc(Test):
+"""
+:avocado: enable
+:avocado: tags=vnc,quick
+"""
+def test_no_vnc(self):
+self.vm.add_args('-nodefaults', '-S')
+self.vm.launch()
+self.assertFalse(self.vm.qmp('query-vnc')['return']['enabled'])
+
+def test_no_vnc_change_password(self):
+self.vm.add_args('-nodefaults', '-S')
+self.vm.launch()
+self.assertFalse(self.vm.qmp('query-vnc')['return']['enabled'])
+set_password_response = self.vm.qmp('change',
+device='vnc',
+target='password',
+arg='new_password')
+self.assertIn('error', set_password_response)
+self.assertEqual(set_password_response['error']['class'],
+ 'GenericError')
+self.assertEqual(set_password_response['error']['desc'],
+ 'Could not set password')
+
+def test_vnc_change_password_requires_a_password(self):
+self.vm.add_args('-nodefaults', '-S', '-vnc', ':0')
+self.vm.launch()
+self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled'])
+set_password_response = self.vm.qmp('change',
+device='vnc',
+target='password',
+arg='new_password')
+self.assertIn('error', set_password_response)
+self.assertEqual(set_password_response['error']['class'],
+ 'GenericError')
+self.assertEqual(set_password_response['error']['desc'],
+ 'Could not set password')
+
+def test_vnc_change_password(self):
+self.vm.add_args('-nodefaults', '-S', '-vnc', ':0,password')
+self.vm.launch()
+self.assertTrue(self.vm.qmp('query-vnc')['return']['enabled'])
+set_password_response = self.vm.qmp('change',
+device='vnc',
+target='password',
+arg='new_password')
+self.assertEqual(set_password_response['return'], {})
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 7/7] configure: Enable out-of-tree acceptance tests

[Eduardo Habkost]

From: Philippe Mathieu-Daudé 

Currently to run Avocado acceptance tests in an out-of-tree
build directory, we need to use the full path to the test:

  build_dir$ avocado run 
/full/path/to/sources/qemu/tests/acceptance/boot_linux_console.py

This patch adds a symlink in the build tree to simplify the
tests invocation, allowing the same command than in in-tree builds:

  build_dir$ avocado run tests/acceptance/boot_linux_console.py

Signed-off-by: Philippe Mathieu-Daudé 
Message-Id: <20180612173437.14462-1-f4...@amsat.org>
Signed-off-by: Eduardo Habkost 
---
 configure | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/configure b/configure
index a8c4094c87..d095ed45a4 100755
--- a/configure
+++ b/configure
@@ -7239,9 +7239,11 @@ for rom in seabios vgabios ; do
 done
 
 # set up tests data directory
-if [ ! -e tests/data ]; then
-symlink "$source_path/tests/data" tests/data
-fi
+for tests_subdir in acceptance data; do
+if [ ! -e tests/$tests_subdir ]; then
+symlink "$source_path/tests/$tests_subdir" tests/$tests_subdir
+fi
+done
 
 # set up qemu-iotests in this build directory
 iotests_common_env="tests/qemu-iotests/common.env"
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 1/7] Remove COPYING.PYTHON

[Eduardo Habkost]

The COPYING.PYTHON file was added when we added the compatibility
argparse.py module, which was licensed under the Python Software
Foundation License Version 2.

Now the compatibility argparse.py module was removed, and we are
not carrying any code under that license anymore.  Remove
COPYING.PYTHON.

Signed-off-by: Eduardo Habkost 
Message-Id: <20180611180152.2681-1-ehabk...@redhat.com>
Reviewed-by: Daniel P. Berrangé 
Reviewed-by: Stefan Hajnoczi 
Signed-off-by: Eduardo Habkost 
---
 COPYING.PYTHON | 270 -
 1 file changed, 270 deletions(-)
 delete mode 100644 COPYING.PYTHON

diff --git a/COPYING.PYTHON b/COPYING.PYTHON
deleted file mode 100644
index 4d3f1ef276..00
--- a/COPYING.PYTHON
+++ /dev/null
@@ -1,270 +0,0 @@
-A. HISTORY OF THE SOFTWARE
-==
-
-Python was created in the early 1990s by Guido van Rossum at Stichting
-Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands
-as a successor of a language called ABC.  Guido remains Python's
-principal author, although it includes many contributions from others.
-
-In 1995, Guido continued his work on Python at the Corporation for
-National Research Initiatives (CNRI, see http://www.cnri.reston.va.us)
-in Reston, Virginia where he released several versions of the
-software.
-
-In May 2000, Guido and the Python core development team moved to
-BeOpen.com to form the BeOpen PythonLabs team.  In October of the same
-year, the PythonLabs team moved to Digital Creations (now Zope
-Corporation, see http://www.zope.com).  In 2001, the Python Software
-Foundation (PSF, see http://www.python.org/psf/) was formed, a
-non-profit organization created specifically to own Python-related
-Intellectual Property.  Zope Corporation is a sponsoring member of
-the PSF.
-
-All Python releases are Open Source (see http://www.opensource.org for
-the Open Source Definition).  Historically, most, but not all, Python
-releases have also been GPL-compatible; the table below summarizes
-the various releases.
-
-Release Derived YearOwner   GPL-
-fromcompatible? (1)
-
-0.9.0 thru 1.2  1991-1995   CWI yes
-1.3 thru 1.5.2  1.2 1995-1999   CNRIyes
-1.6 1.5.2   2000CNRIno
-2.0 1.6 2000BeOpen.com  no
-1.6.1   1.6 2001CNRIyes (2)
-2.1 2.0+1.6.1   2001PSF no
-2.0.1   2.0+1.6.1   2001PSF yes
-2.1.1   2.1+2.0.1   2001PSF yes
-2.2 2.1.1   2001PSF yes
-2.1.2   2.1.1   2002PSF yes
-2.1.3   2.1.2   2002PSF yes
-2.2.1   2.2 2002PSF yes
-2.2.2   2.2.1   2002PSF yes
-2.2.3   2.2.2   2003PSF yes
-2.3 2.2.2   2002-2003   PSF yes
-2.3.1   2.3 2002-2003   PSF yes
-2.3.2   2.3.1   2002-2003   PSF yes
-2.3.3   2.3.2   2002-2003   PSF yes
-2.3.4   2.3.3   2004PSF yes
-2.3.5   2.3.4   2005PSF yes
-2.4 2.3 2004PSF yes
-2.4.1   2.4 2005PSF yes
-2.4.2   2.4.1   2005PSF yes
-2.4.3   2.4.2   2006PSF yes
-2.5 2.4 2006PSF yes
-2.7 2.6 2010PSF yes
-
-Footnotes:
-
-(1) GPL-compatible doesn't mean that we're distributing Python under
-the GPL.  All Python licenses, unlike the GPL, let you distribute
-a modified version without making your changes open source.  The
-GPL-compatible licenses make it possible to combine Python with
-other software that is released under the GPL; the others don't.
-
-(2) According to Richard Stallman, 1.6.1 is not GPL-compatible,
-because its license has a choice of law clause.  According to
-CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1
-is "not incompatible" with the GPL.
-
-Thanks to the many outside volunteers who have worked under Guido's
-direction to make these releases possible.
-
-
-B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON
-===
-
-PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
-
-
-1. This LICENSE AGREEMENT is between the Python Software Foundation
-("PSF"), and the Individual or Organization ("Licensee") accessing and
-otherwise using this software ("Python") in source or binary form and
-its associated documentation.
-
-2. Subject 

[Qemu-devel] [PULL 3/7] scripts/qemu.py: allow adding to the list of extra arguments

[Eduardo Habkost]

From: Cleber Rosa 

Tests will often need to add extra arguments to QEMU command
line arguments.

Signed-off-by: Cleber Rosa 
Reviewed-by: Stefan Hajnoczi 
Message-Id: <20180530184156.15634-3-cr...@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé 
Tested-by: Philippe Mathieu-Daudé 
Signed-off-by: Eduardo Habkost 
---
 scripts/qemu.py | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 08a3e9af5a..7cd8193df8 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -359,3 +359,9 @@ class QEMUMachine(object):
 of the qemu process.
 '''
 return self._iolog
+
+def add_args(self, *args):
+'''
+Adds to the list of extra arguments to be given to the QEMU binary
+'''
+self._args.extend(args)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 0/7] Python queue, 2018-06-15

[Eduardo Habkost]

The following changes since commit 2ef2f16781af9dee6ba6517755e9073ba5799fa2:

  Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20180615a' 
into staging (2018-06-15 18:13:35 +0100)

are available in the Git repository at:

  git://github.com/ehabkost/qemu.git tags/python-next-pull-request

for you to fetch changes up to 1b145d59b74a5880d82954ee940a219e73851f9e:

  configure: Enable out-of-tree acceptance tests (2018-06-15 16:10:11 -0300)


Python queue, 2018-06-15

* Add avocado_qemu: functional/acceptance test infrastructure



Cleber Rosa (5):
  Add functional/acceptance tests infrastructure
  scripts/qemu.py: allow adding to the list of extra arguments
  Acceptance tests: add quick VNC tests
  scripts/qemu.py: introduce set_console() method
  Acceptance tests: add Linux kernel boot and console checking test

Eduardo Habkost (1):
  Remove COPYING.PYTHON

Philippe Mathieu-Daudé (1):
  configure: Enable out-of-tree acceptance tests

 docs/devel/testing.rst| 192 +++
 configure |   8 +-
 scripts/qemu.py   | 103 -
 COPYING.PYTHON| 270 --
 tests/acceptance/README.rst   |  10 +
 tests/acceptance/avocado_qemu/__init__.py |  54 +
 tests/acceptance/boot_linux_console.py|  47 
 tests/acceptance/version.py   |  24 ++
 tests/acceptance/vnc.py   |  60 +
 9 files changed, 494 insertions(+), 274 deletions(-)
 delete mode 100644 COPYING.PYTHON
 create mode 100644 tests/acceptance/README.rst
 create mode 100644 tests/acceptance/avocado_qemu/__init__.py
 create mode 100644 tests/acceptance/boot_linux_console.py
 create mode 100644 tests/acceptance/version.py
 create mode 100644 tests/acceptance/vnc.py

-- 
2.18.0.rc1.1.g3f1ff2140

Re: [Qemu-devel] [PATCH] docs: add guidance on configuring CPU models for x86

[Eduardo Habkost]

On Fri, Jun 15, 2018 at 03:16:09PM +0100, Daniel P. Berrangé wrote:
> With the recent set of CPU hardware vulnerabilities on x86, it is
> increasingly difficult to understand which CPU configurations are
> good to use and what flaws they might be vulnerable to.
> 
> This doc attempts to help management applications and administrators in
> picking sensible CPU configuration on x86 hosts. It outlines which of
> the named CPU models are good choices, and describes which extra CPU
> flags should be enabled to allow the guest to mitigate hardware flaws.
> 
> Signed-off-by: Daniel P. Berrangé 

Thank you very much.

I wonder how we could reference this from the man page and
qemu-doc.html without saying "look at the QEMU source tree".

(I'm not willing to rewrite this using texinfo to make that
happen.)


-- 
Eduardo

Re: [Qemu-devel] Abandon our QMP first policy? (was: [PATCH v3 5/7] hmp: Add info commands for preconfig)

[Eduardo Habkost]

On Fri, Jun 15, 2018 at 06:10:48PM +0200, Markus Armbruster wrote:
> "Dr. David Alan Gilbert"  writes:
> 
> > * Markus Armbruster (arm...@redhat.com) wrote:
> >> Gerd Hoffmann  writes:
> >> 
> >> >   Hi,
> >> >
> >> >> > Now let's review the three commands:
> >> >> > 
> >> >> > * Gerd, why does "info usbhost" have no QMP equivalent?
> >> >
> >> > Works only when running qemu directly, in the libvirt sandbox qemu
> >> > hasn't the permissions needed to scan the host usb bus so that would be
> >> > rather pointless ...
> >> 
> >> I don't think this meets either of the two criteria:
> >> 
> >> * I meets "makes no sense in QMP" only if QMP implies "can't scan host
> >>   USB bus".  Libvirt implies it, but QMP doesn't imply libvirt; it's its
> >>   most important, not its sole user.
> >> 
> >> * It meets "of use only for human users" only if we're convinced it's of
> >>   no use to programs.
> >> 
> >>   To avoid speculation and endless arguments about what could or could
> >>   not be of use, we've always stuck to "when in doubt, assume it could
> >>   be of use".
> >> 
> >>   "Libvirt can't use it" falls short.
> >> 
> >>   "Any management application worth anything would deny QEMU the
> >>   capability to scan the USB host bus, and thus wouldn't be able to use
> >>   it" is exactly the argument we intended to avoid.
> >> 
> >> QMP falling short of completeness in relatively unimportant ways like
> >> this one isn't exactly terrible.  The most serious effect is probably
> >> serving as a bad example that leads to further arguments like this one.
> >> These are well worth avoiding, though.
> >
> > Markus:
> >   a) This is a separate discussion; info usbhost has been there for many
> > years;  this patch set doesn't change that.
> 
> Yes, it's separate.
> 
> >   b) From HMP, if someone wants to add a command like 'info usbhost' to
> > make their debugging of USB easy, then HMP is all for that and there's
> > no way I'm going to require QMP implementations for a debug command.
> 
> As I said before, said someone is welcome to make his case for why the
> command is of use only for human users in the commit message.
> 
> If the QMP maintainer gets a say in judging that case, good.
> 
> But if your "no way" means what I have to assume it means, namely "no
> way I'm taking no for an answer", then we're at an impasse.

Is this really an impasse?  Dave says he isn't going to require
QMP implementations of debug commands.  Isn't a debug command
just for humans, by definition?

(Which I think is true in the specific case of "info usbhost",
because it's really just a wrapper around libusb.  I'm 100% sure
QEMU doesn't need to provide a QMP interface for libusb.)



> 
> Evolving HMP without consideration for QMP's mission makes carrying out
> that mission harder.  Specifically, me having to watch for new HMP
> commands bypassing QMP, then judge and maybe debate whether that's okay
> is beyond my capacity.  The debate part in particular.  I expect to fail
> as a QMP maintainer in that case.

What's QMP mission, exactly?

Do we have any signs of HMP evolving without consideration for
QMP's mission, recently?

Would it help if the HMP maintainer makes sure commit messages or
comments document more clearly why a QMP equivalent is not
necessary?


> 
> Avoiding QMP maintenance balloon out of control was one of our reasons
> for the rule "HMP commands must be implemented in terms of QMP".  We pay
> a modest amount of extra developer effort for a much-needed
> simplification of the maintainers' job.

I don't understand why "HMP commands must be implemented in terms
of QMP" helps avoiding QMP maintenance to balloon out of control.
My impression is that it's the opposite.

I fail to see the benefits of that policy.  What exactly is the
goal here?


> 
> Regarding "modest amount of effort": I already told you I'm prepared to
> back that claim with patches whenever requiring QMP blocks HMP patches.
> If that can't shake your conviction that requiring QMP is impractical, I
> figure nothing will.
> 
> Ways out of the impasse:
> 
> 1. Find a more capable maintainer for QMP (MAINTAINERS sections QMP and
> QAPI Schema).  To give you an idea of the work involved, almost 10% of
> the commits in 2.9 touched the latter.
> 
> 2. Split the work differently: make the HMP maintainer responsible for
> keeping QMP complete.

This sounds like a reasonable way to split responsibilities, to
me.


> 
> 3. Abandon QMP's goal to be complete.  Tell management applications to
> use HMP where QMP falls short.

What "complete" means here?  Why would we ever tell management
applications to use HMP?  We can always implement new QMP
commands if management applications need them.


> 
> To be brutally frank, 2. requires more trust in your willingness to care
> about QMP than I have right now.
> 
> By refusing to cooperate, you can force 3.
> 
> The solution most conducive to my personal well-being is probably 1.
> 
> Any other bright ideas?
> 
> Separate 

Re: [Qemu-devel] bug in reopen arch

[Vladimir Sementsov-Ogievskiy]

14.06.2018 13:46, Kevin Wolf wrote:

Am 12.06.2018 um 20:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

Hi all!

I've faced the following problem:

     1. create image with dirty bitmap, a.qcow2 (start qemu and run qmp
     command block-dirty-bitmap-add)

     2. run the following commands:

     qemu-img create -f qcow2 -b a.qcow2 b.qcow2 10M
     qemu-io -c 'write 0 512' b.qcow2
     qemu-img commit b.qcow2

     3. last command fails with the following output:

Formatting 'b.qcow2', fmt=qcow2 size=68719476736 backing_file=a.qcow2
cluster_size=65536 lazy_refcounts=off refcount_bits=16
wrote 512/512 bytes at offset 0
512 bytes, 1 ops; 0.0953 sec (5.243 KiB/sec and 10.4867 ops/sec)
qemu-img: #block397: Failed to make dirty bitmaps writable: Can't update
bitmap directory: Operation not permitted
qemu-img: Block job failed: Operation not permitted

And problem is that children are reopened _after_ parent. But qcow2 reopen
needs write access to its file, to write IN_USE flag to dirty-bitmaps
extension.

I was aware of a different instance of this problem: Assume a qcow2
image with an unknown autoclear flag (so it will be cleared on r/w
open), which is first opened r/o and then reopened r/w. This will fail
because .bdrv_reopen_prepare doesn't have the permissions yet.


Hm.. If I understand correctly qcow2_reopen_prepare doesn't deal with  
autoclear flags, as it doesn't call qcow2_do_open.




Simply changing the order won't fix this because in the r/w -> r/o, the
driver will legitimately flush its caches in .bdrv_reopen_prepare, and
for this it still needs to be able to write.

We may need to have a way for nodes to access both the old and the new
state of their children. I'm not completely sure how to achieve this
best, though.

When I thought only of permissions, the obvious and simple thing to do
was to just get combined permissions for the old and new state, i.e.
'old_perm | new_perm' and 'old_shared & new_shared'. But I don't think
this is actually enough when the child node switches between a r/w and
a r/o file descriptor because even though QEMU's permission system would
allow the write, you still can't successfully write to a r/o file
descriptor.

Kevin


Maybe we want two .bdrv_reopen_prepare: 
.bdrv_reopen_prepare_before_children and 
.bdrv_reopen_prepare_after_children. But to write something in 
reopen_prepare, we need to move bdrv_set_perm from reopen_commit to 
reopen_prepare.. Is it possible?


Now, I've found the following workaround, what do you think about 
something like this as a temporary fix:


diff --git a/include/block/block.h b/include/block/block.h
index e677080c4e..c21392491d 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -266,7 +266,8 @@ BlockDriverState *bdrv_new_open_driver(BlockDriver 
*drv, const char *node_name,

 BlockReopenQueue *bdrv_reopen_queue(BlockReopenQueue *bs_queue,
 BlockDriverState *bs,
 QDict *options, int flags);
-int bdrv_reopen_multiple(AioContext *ctx, BlockReopenQueue *bs_queue, 
Error **errp);

+int bdrv_reopen_multiple(AioContext *ctx, BlockReopenQueue *bs_queue,
+ bool cheat_reopen_rw, Error **errp);
 int bdrv_reopen(BlockDriverState *bs, int bdrv_flags, Error **errp);
 int bdrv_reopen_prepare(BDRVReopenState *reopen_state,
 BlockReopenQueue *queue, Error **errp);
diff --git a/block.c b/block.c
index 50887087f3..9b50828cd2 100644
--- a/block.c
+++ b/block.c
@@ -2988,7 +2988,8 @@ BlockReopenQueue 
*bdrv_reopen_queue(BlockReopenQueue *bs_queue,

  * All affected nodes must be drained between bdrv_reopen_queue() and
  * bdrv_reopen_multiple().
  */
-int bdrv_reopen_multiple(AioContext *ctx, BlockReopenQueue *bs_queue, 
Error **errp)

+int bdrv_reopen_multiple(AioContext *ctx, BlockReopenQueue *bs_queue,
+ bool cheat_reopen_rw, Error **errp)
 {
 int ret = -1;
 BlockReopenQueueEntry *bs_entry, *next;
@@ -3005,6 +3006,14 @@ int bdrv_reopen_multiple(AioContext *ctx, 
BlockReopenQueue *bs_queue, Error **er

 bs_entry->prepared = true;
 }

+    if (cheat_reopen_rw) {
+    /* reverse queue, to reopen children first */
+    QSIMPLEQ_FOREACH_SAFE(bs_entry, bs_queue, entry, next) {
+    QSIMPLEQ_REMOVE(bs_queue, bs_entry, BlockReopenQueueEntry, 
entry);

+    QSIMPLEQ_INSERT_HEAD(bs_queue, bs_entry, entry);
+    }
+    }
+
 /* If we reach this point, we have success and just need to apply the
  * changes
  */
@@ -3036,11 +3045,13 @@ int bdrv_reopen(BlockDriverState *bs, int 
bdrv_flags, Error **errp)

 int ret = -1;
 Error *local_err = NULL;
 BlockReopenQueue *queue;
+    bool cheat_reopen_rw = bdrv_is_read_only(bs) && (bdrv_flags & 
BDRV_O_RDWR);


 bdrv_subtree_drained_begin(bs);

 queue = bdrv_reopen_queue(NULL, bs, NULL, bdrv_flags);
-    ret = bdrv_reopen_multiple(bdrv_get_aio_context(bs), queue, 
_err);