Re: [Qemu-devel] [PATCH v8 0/5] hypertrace: Lightweight guest-to-QEMU trace channel

2017-08-31 Thread Lluís Vilanova
Stefan Hajnoczi writes:

> On Fri, Aug 25, 2017 at 08:34:54PM -0400, Emilio G. Cota wrote:
>> On Sun, Jul 30, 2017 at 17:08:18 +0300, Lluís Vilanova wrote:
>> > The hypertrace channel allows guest code to emit events in QEMU (the host) 
>> > using
>> > its tracing infrastructure (see "docs/trace.txt"). This works in both 
>> > 'system'
>> > and 'user' modes, is architecture-agnostic and introduces minimal noise on 
>> > the
>> > guest.
>> > 
>> > See first commit for a full description, use-cases and an example.
>> > 
>> > Signed-off-by: Lluís Vilanova 
>> 
>> This would be indeed very useful once TCG instrumentation is in place.
>> 
>> However, I'm not very excited about this being PCI-only and Linux-only for
>> system mode
>> I wonder how we could make this work on all hosts -- did you consider using
>> "magic" instructions? We'd need a different magic instruction for each
>> guest ISA, but the library would hide that anyway (and the library code
>> would be the same for user and system modes).

> Magic instructions can potentially be implemented more efficiently in
> TCG too.  They wouldn't work under KVM/HAX/hvf accelerators though.

Yes, we discussed this on the first version of this series long ago. The special
device makes it work on both TCG and KVM (if it were possible to implement a
TCG-to/from-KVM mode switch that'd be awesome).

Also, both approaches have negligible performance impact on the *guest* time.


Cheers,
  Lluis



Re: [Qemu-devel] [PATCH v8 0/5] hypertrace: Lightweight guest-to-QEMU trace channel

2017-08-31 Thread Stefan Hajnoczi
On Fri, Aug 25, 2017 at 08:34:54PM -0400, Emilio G. Cota wrote:
> On Sun, Jul 30, 2017 at 17:08:18 +0300, Lluís Vilanova wrote:
> > The hypertrace channel allows guest code to emit events in QEMU (the host) 
> > using
> > its tracing infrastructure (see "docs/trace.txt"). This works in both 
> > 'system'
> > and 'user' modes, is architecture-agnostic and introduces minimal noise on 
> > the
> > guest.
> > 
> > See first commit for a full description, use-cases and an example.
> > 
> > Signed-off-by: Lluís Vilanova 
> 
> This would be indeed very useful once TCG instrumentation is in place.
> 
> However, I'm not very excited about this being PCI-only and Linux-only for
> system mode
> I wonder how we could make this work on all hosts -- did you consider using
> "magic" instructions? We'd need a different magic instruction for each
> guest ISA, but the library would hide that anyway (and the library code
> would be the same for user and system modes).

Magic instructions can potentially be implemented more efficiently in
TCG too.  They wouldn't work under KVM/HAX/hvf accelerators though.

Stefan



Re: [Qemu-devel] [PATCH v8 0/5] hypertrace: Lightweight guest-to-QEMU trace channel

2017-08-25 Thread Emilio G. Cota
On Sun, Jul 30, 2017 at 17:08:18 +0300, Lluís Vilanova wrote:
> The hypertrace channel allows guest code to emit events in QEMU (the host) 
> using
> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> and 'user' modes, is architecture-agnostic and introduces minimal noise on the
> guest.
> 
> See first commit for a full description, use-cases and an example.
> 
> Signed-off-by: Lluís Vilanova 

This would be indeed very useful once TCG instrumentation is in place.

However, I'm not very excited about this being PCI-only and Linux-only for
system mode
I wonder how we could make this work on all hosts -- did you consider using
"magic" instructions? We'd need a different magic instruction for each
guest ISA, but the library would hide that anyway (and the library code
would be the same for user and system modes).

Thanks,

Emilio



Re: [Qemu-devel] [PATCH v8 0/5] hypertrace: Lightweight guest-to-QEMU trace channel

2017-08-08 Thread Stefan Hajnoczi
On Fri, Aug 04, 2017 at 09:32:25PM +0300, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > On Sun, Jul 30, 2017 at 05:08:18PM +0300, Lluís Vilanova wrote:
> >> The hypertrace channel allows guest code to emit events in QEMU (the host) 
> >> using
> >> its tracing infrastructure (see "docs/trace.txt"). This works in both 
> >> 'system'
> >> and 'user' modes, is architecture-agnostic and introduces minimal noise on 
> >> the
> >> guest.
> >> 
> >> See first commit for a full description, use-cases and an example.
> >> 
> >> Signed-off-by: Lluís Vilanova 
> >> ---
> >> 
> >> Changes in v8
> >> =
> >> 
> >> * Do not use 'seq' when there's no extra hypertrace arguments (BSD behaves
> >> differently for "seq 0").
> >> * Fix compilation for bsd-user.
> 
> > Hi Lluís,
> > Any changes regarding the fundamental approach since September 2016?
> 
> > Back then I had the following concerns about hypertrace for full-system
> > virtualization:
> 
> > Going to QEMU for every guest trace event has high overhead under
> > virtualization.  The alternative approach is recording separate traces
> > and merging them for analysis.  This is possible with trace-cmd [1] and
> > LTTng [2].
> 
> > Merging traces eliminates the performance bottleneck and does not
> > require new paravirt interfaces or guest tracing libraries.  I think it
> > it would be a distraction to support hypertrace for the virtualization
> > use case because it fundamentally has a high overhead.
> 
> > I see promise in using hypertrace for TCG because it is low-overhead
> > when running inside the QEMU process.  I'll review the patches again
> > with this in mind and not focus on virtualization.
> 
> > [1] https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html
> > [2] 
> > http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Virtual-Machine-Analysis.html
> > and also generic trace synchronization
> > 
> > http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-synchronization.html#Trace_synchronization
> 
> There's been no fundamental changes since then (just the few bits listed in 
> the
> v5-v8 changelog).
> 
> But I'm kind of split on this one.
> 
> If you want high-performance trace correlation, this will work much better for
> TCG than virtualization (where [1] will probably be more efficient).
> 
> If you want a hook to trigger other operations (like in the docs example), I
> think this is the right approach. In fact, my initial interest in hypertrace 
> was
> for instrumentation, so maybe this should be subsumed into the proposal of a
> stable instrumentation API.
> 
> What do you think?

That sounds good.

Stefan


signature.asc
Description: PGP signature


Re: [Qemu-devel] [PATCH v8 0/5] hypertrace: Lightweight guest-to-QEMU trace channel

2017-08-04 Thread Lluís Vilanova
Stefan Hajnoczi writes:

> On Sun, Jul 30, 2017 at 05:08:18PM +0300, Lluís Vilanova wrote:
>> The hypertrace channel allows guest code to emit events in QEMU (the host) 
>> using
>> its tracing infrastructure (see "docs/trace.txt"). This works in both 
>> 'system'
>> and 'user' modes, is architecture-agnostic and introduces minimal noise on 
>> the
>> guest.
>> 
>> See first commit for a full description, use-cases and an example.
>> 
>> Signed-off-by: Lluís Vilanova 
>> ---
>> 
>> Changes in v8
>> =
>> 
>> * Do not use 'seq' when there's no extra hypertrace arguments (BSD behaves
>> differently for "seq 0").
>> * Fix compilation for bsd-user.

> Hi Lluís,
> Any changes regarding the fundamental approach since September 2016?

> Back then I had the following concerns about hypertrace for full-system
> virtualization:

> Going to QEMU for every guest trace event has high overhead under
> virtualization.  The alternative approach is recording separate traces
> and merging them for analysis.  This is possible with trace-cmd [1] and
> LTTng [2].

> Merging traces eliminates the performance bottleneck and does not
> require new paravirt interfaces or guest tracing libraries.  I think it
> it would be a distraction to support hypertrace for the virtualization
> use case because it fundamentally has a high overhead.

> I see promise in using hypertrace for TCG because it is low-overhead
> when running inside the QEMU process.  I'll review the patches again
> with this in mind and not focus on virtualization.

> [1] https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html
> [2] 
> http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Virtual-Machine-Analysis.html
> and also generic trace synchronization
> 
> http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-synchronization.html#Trace_synchronization

There's been no fundamental changes since then (just the few bits listed in the
v5-v8 changelog).

But I'm kind of split on this one.

If you want high-performance trace correlation, this will work much better for
TCG than virtualization (where [1] will probably be more efficient).

If you want a hook to trigger other operations (like in the docs example), I
think this is the right approach. In fact, my initial interest in hypertrace was
for instrumentation, so maybe this should be subsumed into the proposal of a
stable instrumentation API.

What do you think?

Cheers,
  Lluis


> Stefan

>> Changes in v7
>> =
>> 
>> * Use 'expr' instead of assuming 'bash' when generating the "emit.c" file.
>> * Restore generation of trace-events-all.
>> 
>> 
>> Changes in v6
>> =
>> 
>> * Fix compilation errors.
>> 
>> 
>> Changes in v5
>> =
>> 
>> * Rebase on 5a477a7806.
>> * Fix typo in "bsd-user/main.c" [Stephan Hajnoczi].
>> * Replace abort() with exit() in command-line errors [Stephan Hajnoczi].
>> * Fix alignment of data and control channels [Stephan Hajnoczi].
>> * Fix signal reflection in user-mode (SIGINT, SIGABRT, SIGSEGV) [Stephan 
>> Hajnoczi].
>> * Clarify semantics of hypertrace_guest_mmap_check() [Stephan Hajnoczi].
>> * Use uintptr_t instead of unsigned long in SEGV handler [Stephan Hajnoczi].
>> * Emit hypertrace's event with host-endian arguments [Stephan Hajnoczi].
>> * Enable true concurrency between user-mode guest threads by using a 
>> spearate control channel page per client [Stephan Hajnoczi].
>> * Remove unused PAGE_SIZE define [Stephan Hajnoczi].
>> * Submit linux kernel API module separately to Linux upstream [Stephan 
>> Hajnoczi].
>> * Assume guest code events are always enabled.
>> 
>> 
>> Changes in v4
>> =
>> 
>> * Fix typo in stap script example.
>> * Fix compilation instructions in doc/hypertrace.txt.
>> * Rebase on 0737f32daf.
>> 
>> 
>> Changes in v3
>> =
>> 
>> * Rebase on 4a58f35.
>> * Remove debugging printf's.
>> * Fix style issues identified by checkpatch.
>> * Fix control channel mapping in guest linux module.
>> * Add a short event description in "trace-events".
>> * Polish documentation in 1st patch.
>> 
>> 
>> Changes in v2
>> =
>> 
>> * Remove unnecessary casts for g2h() [Eric Blake].
>> * Use perror() [Eric Blake].
>> * Avoid expansions in application example [Eric Blake].
>> * Add copyright in document "hypertrace.txt" [Eric Blake].
>> * Make the user-mode hypertrace invocations thread-safe [Stefan Hajnoczi].
>> * Split dynamic hypertrace configuration into a separate "config" channel.
>> 
>> Lluís Vilanova (5):
>> hypertrace: Add documentation
>> hypertrace: Add tracing event "guest_hypertrace"
>> hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event
>> hypertrace: [softmmu] Add QEMU-side proxy to "guest_hypertrace" event
>> hypertrace: Add guest-side user-level library
>> 
>> 
>> Makefile   |   11 +
>> Makefile.objs  |6 +
>> 

Re: [Qemu-devel] [PATCH v8 0/5] hypertrace: Lightweight guest-to-QEMU trace channel

2017-08-04 Thread Stefan Hajnoczi
On Sun, Jul 30, 2017 at 05:08:18PM +0300, Lluís Vilanova wrote:
> The hypertrace channel allows guest code to emit events in QEMU (the host) 
> using
> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> and 'user' modes, is architecture-agnostic and introduces minimal noise on the
> guest.
> 
> See first commit for a full description, use-cases and an example.
> 
> Signed-off-by: Lluís Vilanova 
> ---
> 
> Changes in v8
> =
> 
> * Do not use 'seq' when there's no extra hypertrace arguments (BSD behaves
>   differently for "seq 0").
> * Fix compilation for bsd-user.

Hi Lluís,
Any changes regarding the fundamental approach since September 2016?

Back then I had the following concerns about hypertrace for full-system
virtualization:

Going to QEMU for every guest trace event has high overhead under
virtualization.  The alternative approach is recording separate traces
and merging them for analysis.  This is possible with trace-cmd [1] and
LTTng [2].

Merging traces eliminates the performance bottleneck and does not
require new paravirt interfaces or guest tracing libraries.  I think it
it would be a distraction to support hypertrace for the virtualization
use case because it fundamentally has a high overhead.

I see promise in using hypertrace for TCG because it is low-overhead
when running inside the QEMU process.  I'll review the patches again
with this in mind and not focus on virtualization.

[1] https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html
[2] 
http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Virtual-Machine-Analysis.html
and also generic trace synchronization

http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/Trace-synchronization.html#Trace_synchronization

Stefan

> Changes in v7
> =
> 
> * Use 'expr' instead of assuming 'bash' when generating the "emit.c" file.
> * Restore generation of trace-events-all.
> 
> 
> Changes in v6
> =
> 
> * Fix compilation errors.
> 
> 
> Changes in v5
> =
> 
> * Rebase on 5a477a7806.
> * Fix typo in "bsd-user/main.c" [Stephan Hajnoczi].
> * Replace abort() with exit() in command-line errors [Stephan Hajnoczi].
> * Fix alignment of data and control channels [Stephan Hajnoczi].
> * Fix signal reflection in user-mode (SIGINT, SIGABRT, SIGSEGV) [Stephan 
> Hajnoczi].
> * Clarify semantics of hypertrace_guest_mmap_check() [Stephan Hajnoczi].
> * Use uintptr_t instead of unsigned long in SEGV handler [Stephan Hajnoczi].
> * Emit hypertrace's event with host-endian arguments [Stephan Hajnoczi].
> * Enable true concurrency between user-mode guest threads by using a spearate 
> control channel page per client [Stephan Hajnoczi].
> * Remove unused PAGE_SIZE define [Stephan Hajnoczi].
> * Submit linux kernel API module separately to Linux upstream [Stephan 
> Hajnoczi].
> * Assume guest code events are always enabled.
> 
> 
> Changes in v4
> =
> 
> * Fix typo in stap script example.
> * Fix compilation instructions in doc/hypertrace.txt.
> * Rebase on 0737f32daf.
> 
> 
> Changes in v3
> =
> 
> * Rebase on 4a58f35.
> * Remove debugging printf's.
> * Fix style issues identified by checkpatch.
> * Fix control channel mapping in guest linux module.
> * Add a short event description in "trace-events".
> * Polish documentation in 1st patch.
> 
> 
> Changes in v2
> =
> 
> * Remove unnecessary casts for g2h() [Eric Blake].
> * Use perror() [Eric Blake].
> * Avoid expansions in application example [Eric Blake].
> * Add copyright in document "hypertrace.txt" [Eric Blake].
> * Make the user-mode hypertrace invocations thread-safe [Stefan Hajnoczi].
> * Split dynamic hypertrace configuration into a separate "config" channel.
> 
> Lluís Vilanova (5):
>   hypertrace: Add documentation
>   hypertrace: Add tracing event "guest_hypertrace"
>   hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event
>   hypertrace: [softmmu] Add QEMU-side proxy to "guest_hypertrace" event
>   hypertrace: Add guest-side user-level library
> 
> 
>  Makefile   |   11 +
>  Makefile.objs  |6 +
>  bsd-user/main.c|   17 +
>  bsd-user/mmap.c|   15 +
>  bsd-user/qemu.h|3 
>  bsd-user/syscall.c |   34 ++-
>  configure  |   36 +++
>  docs/devel/tracing.txt |3 
>  docs/hypertrace.txt|  225 
>  hypertrace/Makefile.objs   |   25 ++
>  hypertrace/common.c|   55 +
>  hypertrace/common.h|   25 ++
>  hypertrace/guest/Makefile  |   30 +++
>  hypertrace/guest/common.c  |  301 ++
>  hypertrace/guest/qemu-hypertrace.h |   80 +++
>  hypertrace/softmmu.c   |  237 

Re: [Qemu-devel] [PATCH v8 0/5] hypertrace: Lightweight guest-to-QEMU trace channel

2017-07-30 Thread no-reply
Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Qemu-devel] [PATCH v8 0/5] hypertrace: Lightweight guest-to-QEMU 
trace channel
Message-id: 150142369849.12995.11229612194223213120.st...@frigg.lan
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
failed=1
echo
fi
n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
b727925815 hypertrace: Add guest-side user-level library
bb83766b6c hypertrace: [softmmu] Add QEMU-side proxy to "guest_hypertrace" event
b1e07ad5aa hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event
add0b54fc3 hypertrace: Add tracing event "guest_hypertrace"
cecce1a6d5 hypertrace: Add documentation

=== OUTPUT BEGIN ===
Checking PATCH 1/5: hypertrace: Add documentation...
Checking PATCH 2/5: hypertrace: Add tracing event "guest_hypertrace"...
Checking PATCH 3/5: hypertrace: [*-user] Add QEMU-side proxy to 
"guest_hypertrace" event...
Checking PATCH 4/5: hypertrace: [softmmu] Add QEMU-side proxy to 
"guest_hypertrace" event...
Checking PATCH 5/5: hypertrace: Add guest-side user-level library...
WARNING: architecture specific defines should be avoided
#171: FILE: hypertrace/guest/common.c:84:
+#if !defined(CONFIG_USER_ONLY) && defined(__linux__)

ERROR: consider using qemu_strtoull in preference to strtoull
#191: FILE: hypertrace/guest/common.c:104:
+uint64_t vv = strtoull(v, , 16);

total: 1 errors, 1 warnings, 432 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-de...@freelists.org