Re: [Qemu-devel] [PATCH v3 19/50] tcg: let plugins instrument memory accesses

[Aaron Lindsay OS via Qemu-devel]

On Jul 01 16:00, Alex Bennée wrote:
> Aaron Lindsay OS  writes:
> > - a way for a plugin to reset any instrumentation decisions made in the
> >   past (essentially calls `tb_flush(cpu);` under the covers). We found
> >   this critical for plugins which undergo state changes during the
> >   course of their execution (i.e. watch for event X, then go into a more
> >   detailed profiling mode until you see event Y)
> 
> check:
> 
> /**
>  * qemu_plugin_reset() - Reset a plugin
>  * @id: this plugin's opaque ID
>  * @cb: callback to be called once the plugin has been reset
>  *
>  * Unregisters all callbacks for the plugin given by @id.
>  *
>  * Do NOT assume that the plugin has been reset once this function returns.
>  * Plugins are reset asynchronously, and therefore the given plugin receives
>  * callbacks until @cb is called.
>  */
> void qemu_plugin_reset(qemu_plugin_id_t id, qemu_plugin_simple_cb_t cb);

Is this essentially synchronous for the current cpu, and only
asynchronous for any other running cpus that didn't trigger the callback
from which the call to qemu_plugin_reset() is being made? If not, could
the state resetting be made synchronous for the current cpu (even if the
callback doesn't happen until the others are complete)? This isn't
absolutely critical, but it is often nice to begin capturing precisely
when you mean to.

> > - the ability for a plugin to trigger a checkpoint to be taken
> 
> We don't have this at the moment. Pranith also mentioned it in his
> review comments. I can see its use but I suspect it won't make the
> initial implementation given the broader requirements of QEMU to do
> checkpointing and how to cleanly expose that to plugins.

Sure. Our patch works for us, but I know we're ignoring a few things
that we can externally ensure won't happen while we're attempting a
checkpoint (i.e. migration) that may have to be considered for something
upstream.

-Aaron

Re: [Qemu-devel] [PATCH v3 19/50] tcg: let plugins instrument memory accesses

[Alex Bennée]

Aaron Lindsay OS  writes:

> On Jun 28 21:52, Alex Bennée wrote:
>> Aaron Lindsay OS  writes:
>> > To make sure I understand - you're implying that one such query will
>> > return the PA from the guest's perspective, right?
>>
>> Yes - although it will be two queries:
>>
>>   struct qemu_plugin_hwaddr *hw = qemu_plugin_get_hwaddr(info, vaddr);
>>
>> This does the actual lookup and stores enough information for the
>> further queries.
>>
>>   uint64_t pa = qemu_plugin_hwaddr_to_raddr(hw);
>>
>> will return the physical address (assuming it's a RAM reference and not
>> some IO location).
>
> Sounds good, as long as we have a good way to either prevent or cleanly
> detect the failure mode for the IO accesses.
>
>> > In terms of our use case - we use QEMU to drive studies to help us
>> > design the next generation of processors. As you can imagine, having the
>> > right physical addresses is important for some aspects of that. We're
>> > currently using a version of Pavel Dovgalyuk's earlier plugin patchset
>> > with some of our own patches/fixes on top, but it would obviously make
>> > our lives easier to work together to get this sort of infrastructure
>> > upstream!
>>
>> Was this:
>>
>>  Date: Tue, 05 Jun 2018 13:39:15 +0300
>>  Message-ID: 
>> <152819515565.30857.16834004920507717324.stgit@pasha-ThinkPad-T60>
>>  Subject: [Qemu-devel] [RFC PATCH v2 0/7] QEMU binary instrumentation 
>> prototype
>
> Yes, that looks like the one.
>
>> What patches did you add on top?
>
> We added:
> - plugin support for linux-user mode (I sent that one upstream, I think)
> - memory tracing support and a VA->PA conversion helper

check

> - a way for a plugin to request getting a callback just before QEMU
>   exits to clean up any internal state

check - qemu_plugin_register_atexit_cb

> - a way for a plugin to reset any instrumentation decisions made in the
>   past (essentially calls `tb_flush(cpu);` under the covers). We found
>   this critical for plugins which undergo state changes during the
>   course of their execution (i.e. watch for event X, then go into a more
>   detailed profiling mode until you see event Y)

check:

/**
 * qemu_plugin_reset() - Reset a plugin
 * @id: this plugin's opaque ID
 * @cb: callback to be called once the plugin has been reset
 *
 * Unregisters all callbacks for the plugin given by @id.
 *
 * Do NOT assume that the plugin has been reset once this function returns.
 * Plugins are reset asynchronously, and therefore the given plugin receives
 * callbacks until @cb is called.
 */
void qemu_plugin_reset(qemu_plugin_id_t id, qemu_plugin_simple_cb_t cb);


> - instrumentation at the TB granularity (in addition to the existing
>   instruction-level support)

check

/**
 * qemu_plugin_register_vcpu_tb_trans_cb() - register a translate cb
 * @id: plugin ID
 * @cb: callback function
 *
 * The @cb function is called every time a translation occurs. The @cb
 * function is passed an opaque qemu_plugin_type which it can query
 * for additional information including the list of translated
 * instructions. At this point the plugin can register further
 * callbacks to be triggered when the block or individual instruction
 * executes.
 */

and then you can have instruction or TB level callbacks:

/**
 * qemu_plugin_register_vcpu_tb_trans_exec_cb() - register execution callback
 * @tb: the opaque qemu_plugin_tb handle for the translation
 * @cb: callback function
 * @flags: does the plugin read or write the CPU's registers?
 * @userdata: any plugin data to pass to the @cb?
 *
 * The @cb function is called every time a translated unit executes.
 */
void qemu_plugin_register_vcpu_tb_exec_cb(struct qemu_plugin_tb *tb,
  qemu_plugin_vcpu_udata_cb_t cb,
  enum qemu_plugin_cb_flags flags,
  void *userdata);

Or the inline equivalent.


> - the ability for a plugin to trigger a checkpoint to be taken

We don't have this at the moment. Pranith also mentioned it in his
review comments. I can see its use but I suspect it won't make the
initial implementation given the broader requirements of QEMU to do
checkpointing and how to cleanly expose that to plugins.

>
> -Aaron


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 19/50] tcg: let plugins instrument memory accesses

[Aaron Lindsay OS via Qemu-devel]

On Jun 28 21:52, Alex Bennée wrote:
> Aaron Lindsay OS  writes:
> > To make sure I understand - you're implying that one such query will
> > return the PA from the guest's perspective, right?
> 
> Yes - although it will be two queries:
> 
>   struct qemu_plugin_hwaddr *hw = qemu_plugin_get_hwaddr(info, vaddr);
> 
> This does the actual lookup and stores enough information for the
> further queries.
> 
>   uint64_t pa = qemu_plugin_hwaddr_to_raddr(hw);
> 
> will return the physical address (assuming it's a RAM reference and not
> some IO location).

Sounds good, as long as we have a good way to either prevent or cleanly
detect the failure mode for the IO accesses.

> > In terms of our use case - we use QEMU to drive studies to help us
> > design the next generation of processors. As you can imagine, having the
> > right physical addresses is important for some aspects of that. We're
> > currently using a version of Pavel Dovgalyuk's earlier plugin patchset
> > with some of our own patches/fixes on top, but it would obviously make
> > our lives easier to work together to get this sort of infrastructure
> > upstream!
> 
> Was this:
> 
>  Date: Tue, 05 Jun 2018 13:39:15 +0300
>  Message-ID: 
> <152819515565.30857.16834004920507717324.stgit@pasha-ThinkPad-T60>
>  Subject: [Qemu-devel] [RFC PATCH v2 0/7] QEMU binary instrumentation 
> prototype

Yes, that looks like the one.

> What patches did you add on top?

We added:
- plugin support for linux-user mode (I sent that one upstream, I think)
- memory tracing support and a VA->PA conversion helper
- a way for a plugin to request getting a callback just before QEMU
  exits to clean up any internal state
- a way for a plugin to reset any instrumentation decisions made in the
  past (essentially calls `tb_flush(cpu);` under the covers). We found
  this critical for plugins which undergo state changes during the
  course of their execution (i.e. watch for event X, then go into a more
  detailed profiling mode until you see event Y)
- instrumentation at the TB granularity (in addition to the existing
  instruction-level support)
- the ability for a plugin to trigger a checkpoint to be taken

-Aaron

Re: [Qemu-devel] [PATCH v3 19/50] tcg: let plugins instrument memory accesses

[Alex Bennée]

Aaron Lindsay OS  writes:

> On Jun 28 18:11, Alex Bennée wrote:
>> Aaron Lindsay OS  writes:
>> > On Jun 14 18:11, Alex Bennée wrote:
>> >> From: "Emilio G. Cota" 
>> >>
>> >> Here the trickiest feature is passing the host address to
>> >> memory callbacks that request it. Perhaps it would be more
>> >> appropriate to pass a "physical" address to plugins, but since
>> >> in QEMU host addr ~= guest physical, I'm going with that for
>> >> simplicity.
>> >
>> > How much more difficult would it be to get the true physical address (on
>> > the guest)?
>>
>> Previously there was a helper that converted host address (i.e. where
>> QEMU actually stores that value) back to the physical address (ram
>> offset + ram base). However the code for calculating all of this is
>> pretty invasive and requires tweaks to all the softmmu TCG backends as
>> well as hooks into a slew of memory functions.
>>
>> I'm re-working this now so we just have the one memory callback and we
>> provide a helper function that can provide an opaque hwaddr struct which
>> can then be queried.
>
> To make sure I understand - you're implying that one such query will
> return the PA from the guest's perspective, right?

Yes - although it will be two queries:

  struct qemu_plugin_hwaddr *hw = qemu_plugin_get_hwaddr(info, vaddr);

This does the actual lookup and stores enough information for the
further queries.

  uint64_t pa = qemu_plugin_hwaddr_to_raddr(hw);

will return the physical address (assuming it's a RAM reference and not
some IO location).

>
>> The catch is you can only call this helper during a
>> memory callback.
>
> Does this mean it will be difficult to get the physical address for the
> bytes containing the instruction encoding itself?

Hmm good question. We track the hostaddr of the instructions as we load
them so we should be able to track that back to the guest physical
address. There isn't a helper for doing that yet though.

>
>> I'm not sure if having this restriction violates our
>> aim of not leaking implementation details to the plugin but it makes the
>> code simpler.
>
> Assuming that the purpose of "not leaking implementation details" is to
> allow the same plugin interface to work with other backend
> implementations in the future, isn't this probably fine?

Quite. We don't want plugin authors to make any assumptions about the
internals of the TCG. It's not totally opaque because there are
translation time events where we offer the plugin a chance to instrument
individual instructions (or even a "block") which obviously exposes
there is a JIT of some sort.

> It may add an
> unnecessary limitation for another backend driving the same plugin
> interface, but I don't think it likely changes the structure of the
> interface itself. And that seems like the sort of restriction that could
> easily be dropped in the future while remaining backwards-compatible.
>
>> Internally what the helper does is simply re-query the SoftMMU TLB. As
>> the TLBs are per-CPU nothing else can have touched the TLB and the cache
>> should be hot so the cost of lookup should be minor. We could also
>> potentially expand the helpers so if you are interested in only IO
>> accesses we can do the full resolution and figure out what device we
>> just accessed.
>
> Oh, so you're already working on doing just what I asked about?

Yes.

>
>> > This is important enough to me that I would be willing to help if
>> > pointed in the right direction.
>>
>> Well I'll certainly CC on the next series (hopefully posted Monday,
>> softfreeze starts Tuesday). I'll welcome any testing and review. Also if
>> you can tell us more about your use case that will help.
>
> Awesome, thanks!
>
> In terms of our use case - we use QEMU to drive studies to help us
> design the next generation of processors. As you can imagine, having the
> right physical addresses is important for some aspects of that. We're
> currently using a version of Pavel Dovgalyuk's earlier plugin patchset
> with some of our own patches/fixes on top, but it would obviously make
> our lives easier to work together to get this sort of infrastructure
> upstream!

Was this:

 Date: Tue, 05 Jun 2018 13:39:15 +0300
 Message-ID: <152819515565.30857.16834004920507717324.stgit@pasha-ThinkPad-T60>
 Subject: [Qemu-devel] [RFC PATCH v2 0/7] QEMU binary instrumentation prototype

There have certainly been a lot of attempts to getting some sort of
plugin functionality into QEMU. I make no promises this one will be the
one but we shall see!

What patches did you add on top?

>
> -Aaron


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 19/50] tcg: let plugins instrument memory accesses

[Aaron Lindsay OS via Qemu-devel]

On Jun 28 18:11, Alex Bennée wrote:
> Aaron Lindsay OS  writes:
> > On Jun 14 18:11, Alex Bennée wrote:
> >> From: "Emilio G. Cota" 
> >>
> >> Here the trickiest feature is passing the host address to
> >> memory callbacks that request it. Perhaps it would be more
> >> appropriate to pass a "physical" address to plugins, but since
> >> in QEMU host addr ~= guest physical, I'm going with that for
> >> simplicity.
> >
> > How much more difficult would it be to get the true physical address (on
> > the guest)?
> 
> Previously there was a helper that converted host address (i.e. where
> QEMU actually stores that value) back to the physical address (ram
> offset + ram base). However the code for calculating all of this is
> pretty invasive and requires tweaks to all the softmmu TCG backends as
> well as hooks into a slew of memory functions.
> 
> I'm re-working this now so we just have the one memory callback and we
> provide a helper function that can provide an opaque hwaddr struct which
> can then be queried.

To make sure I understand - you're implying that one such query will
return the PA from the guest's perspective, right?

> The catch is you can only call this helper during a
> memory callback.

Does this mean it will be difficult to get the physical address for the
bytes containing the instruction encoding itself?

> I'm not sure if having this restriction violates our
> aim of not leaking implementation details to the plugin but it makes the
> code simpler.

Assuming that the purpose of "not leaking implementation details" is to
allow the same plugin interface to work with other backend
implementations in the future, isn't this probably fine? It may add an
unnecessary limitation for another backend driving the same plugin
interface, but I don't think it likely changes the structure of the
interface itself. And that seems like the sort of restriction that could
easily be dropped in the future while remaining backwards-compatible.

> Internally what the helper does is simply re-query the SoftMMU TLB. As
> the TLBs are per-CPU nothing else can have touched the TLB and the cache
> should be hot so the cost of lookup should be minor. We could also
> potentially expand the helpers so if you are interested in only IO
> accesses we can do the full resolution and figure out what device we
> just accessed.

Oh, so you're already working on doing just what I asked about?

> > This is important enough to me that I would be willing to help if
> > pointed in the right direction.
> 
> Well I'll certainly CC on the next series (hopefully posted Monday,
> softfreeze starts Tuesday). I'll welcome any testing and review. Also if
> you can tell us more about your use case that will help.

Awesome, thanks!

In terms of our use case - we use QEMU to drive studies to help us
design the next generation of processors. As you can imagine, having the
right physical addresses is important for some aspects of that. We're
currently using a version of Pavel Dovgalyuk's earlier plugin patchset
with some of our own patches/fixes on top, but it would obviously make
our lives easier to work together to get this sort of infrastructure
upstream!

-Aaron

Re: [Qemu-devel] [PATCH v3 19/50] tcg: let plugins instrument memory accesses

[Alex Bennée]

Aaron Lindsay OS  writes:

> On Jun 14 18:11, Alex Bennée wrote:
>> From: "Emilio G. Cota" 
>>
>> Here the trickiest feature is passing the host address to
>> memory callbacks that request it. Perhaps it would be more
>> appropriate to pass a "physical" address to plugins, but since
>> in QEMU host addr ~= guest physical, I'm going with that for
>> simplicity.
>
> How much more difficult would it be to get the true physical address (on
> the guest)?

Previously there was a helper that converted host address (i.e. where
QEMU actually stores that value) back to the physical address (ram
offset + ram base). However the code for calculating all of this is
pretty invasive and requires tweaks to all the softmmu TCG backends as
well as hooks into a slew of memory functions.

I'm re-working this now so we just have the one memory callback and we
provide a helper function that can provide an opaque hwaddr struct which
can then be queried. The catch is you can only call this helper during a
memory callback. I'm not sure if having this restriction violates our
aim of not leaking implementation details to the plugin but it makes the
code simpler.

Internally what the helper does is simply re-query the SoftMMU TLB. As
the TLBs are per-CPU nothing else can have touched the TLB and the cache
should be hot so the cost of lookup should be minor. We could also
potentially expand the helpers so if you are interested in only IO
accesses we can do the full resolution and figure out what device we
just accessed.

> This is important enough to me that I would be willing to help if
> pointed in the right direction.

Well I'll certainly CC on the next series (hopefully posted Monday,
softfreeze starts Tuesday). I'll welcome any testing and review. Also if
you can tell us more about your use case that will help.

>
> -Aaron


--
Alex Bennée

Re: [Qemu-devel] [PATCH v3 19/50] tcg: let plugins instrument memory accesses

[Aaron Lindsay OS via Qemu-devel]

On Jun 14 18:11, Alex Bennée wrote:
> From: "Emilio G. Cota" 
> 
> Here the trickiest feature is passing the host address to
> memory callbacks that request it. Perhaps it would be more
> appropriate to pass a "physical" address to plugins, but since
> in QEMU host addr ~= guest physical, I'm going with that for
> simplicity.

How much more difficult would it be to get the true physical address (on
the guest)?

This is important enough to me that I would be willing to help if
pointed in the right direction.

-Aaron

Re: [Qemu-devel] [PATCH v3 19/50] tcg: let plugins instrument memory accesses

[Richard Henderson]

On 6/14/19 10:11 AM, Alex Bennée wrote:
> +static inline void set_hostaddr(CPUArchState *env, TCGMemOp mo, void *haddr)
> +{
> +#ifdef CONFIG_PLUGIN
> +if (mo & MO_HADDR) {
> +env_tlb(env)->c.hostaddr = haddr;
> +}
> +#endif
> +}
> +

Even if we weren't talking about recomputing this in the helper, would an
unconditional store be cheaper?


r~

[Qemu-devel] [PATCH v3 19/50] tcg: let plugins instrument memory accesses

[Alex Bennée]

From: "Emilio G. Cota" 

XXX: store hostaddr from non-i386 TCG backends (do it in a helper?)
XXX: what hostaddr to return for I/O accesses?
XXX: what hostaddr to return for cross-page accesses?

Here the trickiest feature is passing the host address to
memory callbacks that request it. Perhaps it would be more
appropriate to pass a "physical" address to plugins, but since
in QEMU host addr ~= guest physical, I'm going with that for
simplicity.

To keep the implementation simple we piggy-back on the TLB fast path,
and thus can only provide the host address _after_ memory accesses
have occurred. For the slow path, it's a bit tedious because there
are many places to update, but it's fairly simple.

However, note that cross-page accesses are tricky, since the
access might be to non-contiguous host addresses. So I'm punting
on that and just passing NULL.

Signed-off-by: Emilio G. Cota 
Signed-off-by: Alex Bennée 

---
v3
  - fixes for cpu_neg()
---
 accel/tcg/atomic_template.h   |  5 +++
 accel/tcg/cpu-exec.c  |  3 ++
 accel/tcg/cputlb.c| 37 +
 accel/tcg/plugin-gen.c| 17 +-
 include/exec/cpu-defs.h   |  9 +
 include/exec/cpu_ldst.h   |  9 +
 include/exec/cpu_ldst_template.h  | 40 ++-
 include/exec/cpu_ldst_useronly_template.h | 34 ---
 tcg/i386/tcg-target.inc.c |  8 +
 tcg/tcg-op.c  | 40 ++-
 tcg/tcg.h |  1 +
 11 files changed, 153 insertions(+), 50 deletions(-)

diff --git a/accel/tcg/atomic_template.h b/accel/tcg/atomic_template.h
index 04c4c7b0d2..33ddfd498c 100644
--- a/accel/tcg/atomic_template.h
+++ b/accel/tcg/atomic_template.h
@@ -18,6 +18,7 @@
  * License along with this library; if not, see .
  */
 
+#include "qemu/plugin.h"
 #include "trace/mem.h"
 
 #if DATA_SIZE == 16
@@ -73,6 +74,8 @@ void atomic_trace_rmw_pre(CPUArchState *env, target_ulong 
addr, uint8_t info)
 static inline void atomic_trace_rmw_post(CPUArchState *env, target_ulong addr,
  void *haddr, uint8_t info)
 {
+qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, haddr, info);
+qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, haddr, info | TRACE_MEM_ST);
 }
 
 static inline
@@ -84,6 +87,7 @@ void atomic_trace_ld_pre(CPUArchState *env, target_ulong 
addr, uint8_t info)
 static inline void atomic_trace_ld_post(CPUArchState *env, target_ulong addr,
 void *haddr, uint8_t info)
 {
+qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, haddr, info);
 }
 
 static inline
@@ -95,6 +99,7 @@ void atomic_trace_st_pre(CPUArchState *env, target_ulong 
addr, uint8_t info)
 static inline void atomic_trace_st_post(CPUArchState *env, target_ulong addr,
 void *haddr, uint8_t info)
 {
+qemu_plugin_vcpu_mem_cb(env_cpu(env), addr, haddr, info);
 }
 #endif /* ATOMIC_TEMPLATE_COMMON */
 
diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
index 6c85c3ee1e..c21353e54f 100644
--- a/accel/tcg/cpu-exec.c
+++ b/accel/tcg/cpu-exec.c
@@ -272,6 +272,7 @@ void cpu_exec_step_atomic(CPUState *cpu)
 qemu_mutex_unlock_iothread();
 }
 assert_no_pages_locked();
+qemu_plugin_disable_mem_helpers(cpu);
 }
 
 if (in_exclusive_region) {
@@ -705,6 +706,8 @@ int cpu_exec(CPUState *cpu)
 if (qemu_mutex_iothread_locked()) {
 qemu_mutex_unlock_iothread();
 }
+qemu_plugin_disable_mem_helpers(cpu);
+
 assert_no_pages_locked();
 }
 
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index 6a0dc438ff..b39c1f06f7 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -879,9 +879,18 @@ static void tlb_fill(CPUState *cpu, target_ulong addr, int 
size,
 assert(ok);
 }
 
+static inline void set_hostaddr(CPUArchState *env, TCGMemOp mo, void *haddr)
+{
+#ifdef CONFIG_PLUGIN
+if (mo & MO_HADDR) {
+env_tlb(env)->c.hostaddr = haddr;
+}
+#endif
+}
+
 static uint64_t io_readx(CPUArchState *env, CPUIOTLBEntry *iotlbentry,
  int mmu_idx, target_ulong addr, uintptr_t retaddr,
- MMUAccessType access_type, int size)
+ TCGMemOp mo, MMUAccessType access_type, int size)
 {
 CPUState *cpu = env_cpu(env);
 hwaddr mr_offset;
@@ -891,6 +900,9 @@ static uint64_t io_readx(CPUArchState *env, CPUIOTLBEntry 
*iotlbentry,
 bool locked = false;
 MemTxResult r;
 
+/* XXX Any sensible choice other than NULL? */
+set_hostaddr(env, mo, NULL);
+
 section = iotlb_to_section(cpu, iotlbentry->addr, iotlbentry->attrs);
 mr = section->mr;
 mr_offset = (iotlbentry->addr & TARGET_PAGE_MASK) + addr;
@@ -925,7 +937,7 @@ static uint64_t io_readx(CPUArchState *env, CPUIOTLBEntry