Pierrick Bouvier <pierrick.bouv...@linaro.org> writes: > On 7/28/25 12:03 PM, Pierrick Bouvier wrote: >> Hi Alex, >> On 7/27/25 1:32 AM, Alex Bennée wrote: >>> As our set of multiarch tests has grown the practice of running every >>> plugin with every test is becoming unsustainable. If we switch to >>> ensuring every test gets run with at least one plugin we can speed >>> things up. >>> >>> Some plugins do need to be run with specific tests (for example the >>> memory instrumentation test). We can handle this by manually adding >>> them to EXTRA_RUNS. We also need to wrap rules in a CONFIG_PLUGIN test >>> so we don't enable the runs when plugins are not enabled. >>> >>> Reviewed-by: Manos Pitsidianakis <manos.pitsidiana...@linaro.org> >>> Signed-off-by: Alex Bennée <alex.ben...@linaro.org> >>> Message-ID: <20250725154517.3523095-12-alex.ben...@linaro.org> >>> >>> diff --git a/tests/tcg/Makefile.target b/tests/tcg/Makefile.target >>> index a12b15637ea..18afd5be194 100644 >>> --- a/tests/tcg/Makefile.target >>> +++ b/tests/tcg/Makefile.target >>> @@ -173,14 +173,25 @@ PLUGINS=$(filter-out $(DISABLE_PLUGINS), \ >>> # We need to ensure expand the run-plugin-TEST-with-PLUGIN >>> # pre-requistes manually here as we can't use stems to handle it. We >>> # only expand MULTIARCH_TESTS which are common on most of our targets >>> -# to avoid an exponential explosion as new tests are added. We also >>> -# add some special helpers the run-plugin- rules can use below. >>> +# and rotate the plugins so we don't grow too out of control as new >>> +# tests are added. Plugins that need to run with a specific test >>> +# should ensure they add their combination to EXTRA_RUNS. >>> ifneq ($(MULTIARCH_TESTS),) >>> -$(foreach p,$(PLUGINS), \ >>> - $(foreach t,$(MULTIARCH_TESTS),\ >>> - $(eval run-plugin-$(t)-with-$(p): $t $p) \ >>> - $(eval RUN_TESTS+=run-plugin-$(t)-with-$(p)))) >>> + >>> +NUM_PLUGINS := $(words $(PLUGINS)) >>> +NUM_TESTS := $(words $(MULTIARCH_TESTS)) >>> + >>> +define mod_plus_one >>> + $(shell $(PYTHON) -c "print( ($(1) % $(2)) + 1 )") >>> +endef >>> + >>> +$(foreach _idx, $(shell seq 1 $(NUM_TESTS)), \ >>> + $(eval _test := $(word $(_idx), $(MULTIARCH_TESTS))) \ >>> + $(eval _plugin := $(word $(call mod_plus_one, $(_idx), $(NUM_PLUGINS)), >>> $(PLUGINS))) \ >>> + $(eval run-plugin-$(_test)-with-$(_plugin): $(_test) $(_plugin)) \ >>> + $(eval RUN_TESTS+=run-plugin-$(_test)-with-$(_plugin))) >>> + >>> endif # MULTIARCH_TESTS >>> endif # CONFIG_PLUGIN >>> diff --git a/tests/tcg/multiarch/Makefile.target >>> b/tests/tcg/multiarch/Makefile.target >>> index bfdf7197a7b..38345ff8805 100644 >>> --- a/tests/tcg/multiarch/Makefile.target >>> +++ b/tests/tcg/multiarch/Makefile.target >>> @@ -189,6 +189,10 @@ run-plugin-semiconsole-with-%: >>> TESTS += semihosting semiconsole >>> endif >>> +test-plugin-mem-access: CFLAGS+=-pthread -O0 >>> +test-plugin-mem-access: LDFLAGS+=-pthread -O0 >>> + >>> +ifeq ($(CONFIG_PLUGIN),y) >>> # Test plugin memory access instrumentation >>> run-plugin-test-plugin-mem-access-with-libmem.so: \ >>> PLUGIN_ARGS=$(COMMA)print-accesses=true >>> @@ -197,8 +201,8 @@ run-plugin-test-plugin-mem-access-with-libmem.so: \ >>> $(SRC_PATH)/tests/tcg/multiarch/check-plugin-output.sh \ >>> $(QEMU) $< >>> -test-plugin-mem-access: CFLAGS+=-pthread -O0 >>> -test-plugin-mem-access: LDFLAGS+=-pthread -O0 >>> +EXTRA_RUNS += run-plugin-test-plugin-mem-access-with-libmem.so >>> +endif >>> # Update TESTS >>> TESTS += $(MULTIARCH_TESTS) >>> diff --git a/tests/tcg/multiarch/system/Makefile.softmmu-target >>> b/tests/tcg/multiarch/system/Makefile.softmmu-target >>> index 5acf2700812..4171b4e6aa0 100644 >>> --- a/tests/tcg/multiarch/system/Makefile.softmmu-target >>> +++ b/tests/tcg/multiarch/system/Makefile.softmmu-target >>> @@ -71,8 +71,11 @@ endif >>> MULTIARCH_RUNS += run-gdbstub-memory run-gdbstub-interrupt \ >>> run-gdbstub-untimely-packet run-gdbstub-registers >>> +ifeq ($(CONFIG_PLUGIN),y) >>> # Test plugin memory access instrumentation >>> -run-plugin-memory-with-libmem.so: \ >>> - PLUGIN_ARGS=$(COMMA)region-summary=true >>> -run-plugin-memory-with-libmem.so: \ >>> - >>> CHECK_PLUGIN_OUTPUT_COMMAND=$(MULTIARCH_SYSTEM_SRC)/validate-memory-counts.py >>> $@.out >>> +run-plugin-memory-with-libmem.so: memory libmem.so >>> +run-plugin-memory-with-libmem.so: PLUGIN_ARGS=$(COMMA)region-summary=true >>> +run-plugin-memory-with-libmem.so: >>> CHECK_PLUGIN_OUTPUT_COMMAND=$(MULTIARCH_SYSTEM_SRC)/validate-memory-counts.py >>> $@.out >>> + >>> +EXTRA_RUNS += run-plugin-memory-with-libmem.so >>> +endif >> I'm not sure how it's related, but check-tcg on aarch64 host now >> fails >> [1] since this series was merged, and I suspect it may be related to >> this patch. I didn't spend time to reproduce and investigate it. >> > > Reverting this patch solved the problem on my side. As well, I'm not > sure if it's a real problem to run all plugins for all tests. At the > moment, it takes 5 min on a slow machine with 4 cpus and a sanitized > build, something we can probably live with for now.
It definitely is - but it really shows up on linux-user builds because we have an increasing number of multiarch tests and its not sustainable to increase by $NARCH * $NPLUGIN everytime a new multiarch test is added. > > Unfortunately Alex is out this week, but it would be nice if another > maintainer could make a PR reverting this patch. > > Regards, > Pierrick > >> [1] >> https://github.com/pbo-linaro/qemu-ci/actions/runs/16575679153/job/46879690693 -- Alex Bennée Virtualisation Tech Lead @ Linaro
