Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Greg Clayton via lldb-dev]

> On Oct 2, 2020, at 3:51 AM, Pavel Labath  wrote:
> 
> On 01/10/2020 22:32, Walter wrote:
>> After a chat with Greg, we agreed on this set of commands
>> 
>> 
>> trace load /path/to/json process trace start/stop process trace save
>> /path/to/json thread trace start/stop thread trace dump [instructions |
>> functions]
>> 
> 
> Thanks. The new commands look good to me.

Great, we can move the "trace dump" over to "thread trace dump" for 
https://reviews.llvm.org/D86670  and keep that 
moving.

> The multi-process trace concept is interesting. I don't question its
> usefulness -- I am sure it can be useful for various kinds of analysis
> (though I've never used that myself). I am wondering though about how to
> represent this thing in lldb, as we don't really have anything close to
> the concept of "debugging" all processes on a given system.
> 
> The only thing that comes close is probably the kernel-level debugging.
> One idea (which has just occurred to me, so it may not be good) might be
> to make these traces behave similarly to that. I.e., create a single
> target/process with one "thread" per physical cpu, and then have a
> special "os plugin" like thing which would present individual
> process/threads.

I don't know enough about how trace data is stored or annotated after the raw 
data is pulled from the cores, but to make it useful it must be able to be 
associated with processes and threads somehow otherwise it would be just a 
bunch of addresses that would all overlap between many processes. 

 
> That would have the advantage of maintaining the one trace-one target
> invariant and also would preserve the information about relative timings
> of individual "processes". I think that wuold be an interesting way to
> view these things, but I don't know if it would be the best one...

I might suggest that each trace plug-in should do its best to represent 
processes and threads as separate entities so that they all remain separate. 
What ever data starts out as should be abstracted and I think I would rather 
see individual processes with their threads if that is possible to do, but I am 
just thinking of this with just a bit of knowledge tracing data. I think many 
chip makers create these trace formats and they are designed from a "trace a 
core" perspective, but if we can tame this data and present it as users would 
want to see it instead of trying to represent it as the data is stored, I think 
we will have a compelling trace feature in our debugger.

Greg___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Pavel Labath via lldb-dev]

On 01/10/2020 22:32, Walter wrote:
> After a chat with Greg, we agreed on this set of commands
> 
> 
> trace load /path/to/json process trace start/stop process trace save
> /path/to/json thread trace start/stop thread trace dump [instructions |
> functions]
> 

Thanks. The new commands look good to me.


The multi-process trace concept is interesting. I don't question its
usefulness -- I am sure it can be useful for various kinds of analysis
(though I've never used that myself). I am wondering though about how to
represent this thing in lldb, as we don't really have anything close to
the concept of "debugging" all processes on a given system.

The only thing that comes close is probably the kernel-level debugging.
One idea (which has just occurred to me, so it may not be good) might be
to make these traces behave similarly to that. I.e., create a single
target/process with one "thread" per physical cpu, and then have a
special "os plugin" like thing which would present individual
process/threads.

That would have the advantage of maintaining the one trace-one target
invariant and also would preserve the information about relative timings
of individual "processes". I think that wuold be an interesting way to
view these things, but I don't know if it would be the best one...

pl
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Greg Clayton via lldb-dev]

I had accepted the patch https://reviews.llvm.org/D86670 
, but then marked as "Request Changes" while 
we discuss the commands in this RFC after new comments came in.


> On Oct 1, 2020, at 1:42 PM, Greg Clayton  wrote:
> 
> We spoke a bit after Panel's comments which made sense and we propose the 
> commands Walter sent below. Let us know what everyone thinks of this 
> organization of the command structure!
> 
>> On Oct 1, 2020, at 1:32 PM, Walter > > wrote:
>> 
>> After a chat with Greg, we agreed on this set of commands
>> 
>> 
>> trace load /path/to/json
>> 
>> process trace start/stop
>> process trace save /path/to/json
>> 
>> thread trace start/stop
>> thread trace dump [instructions | functions]
>> 
>> 
>> Il giorno gio 1 ott 2020 alle ore 13:21 Greg Clayton > > ha scritto:
>> 
>> 
>> > On Oct 1, 2020, at 7:08 AM, Pavel Labath via lldb-dev 
>> > mailto:lldb-dev@lists.llvm.org>> wrote:
>> > 
>> > Thank you for writing this Walter. I think this document will be a
>> > useful reference both now and in the future.
>> > 
>> > The part that's not clear to me is what is the story with multi-process
>> > traces. The file format enables those, but it's not clear how are they
>> > going be created or used. Can you elaborate more on what you intend to
>> > use those for?
>> 
>> Mainly for system trace kinds of things where an entire system gets traced.
>> 
>> > 
>> > The main reason I am asking that is because I am thinking about the
>> > proposed command structure. I'm wondering if it would not be better to
>> > fit this into the existing target/process/thread commands instead of
>> > adding a new top-level command. For example, one could imagine the
>> > following set of commands:
>> > 
>> > - "process trace start" + "thread trace start" instead of "thread trace
>> > [tid]". That would be similar to "process continue" + "thread continue".
>> > - "thread trace dump [tid]" instead of "trace dump [-t tid]". That would
>> > be similar to "thread continue" and other thread control commands.
>> > - "target create --trace" instead of "trace load". (analogous to target
>> > create --core).
>> > - "process trace save" instead of "trace save" -- (mostly) analogous to
>> > "process save-core"
>> 
>> > I am thinking this composition may fit in better into the existing lldb
>> > command landscape, though I also see the appeal in grouping everything
>> > trace-related under a single top-level command. What do you think?
>> > 
>> > The main place where this idea breaks down is the multi-process traces.
>> > While we could certainly make "target create --trace" create multiple
>> > targets, that would be fairly unusual. OTOH, the whole concept of having
>> > multiple targets share something is a pretty unusual thing for lldb.
>> > That's why I'd like to hear more about where you want to go with this idea.
>> 
>> I kind of see tracing has having two sides:
>> 1 - post mortem tracing for individual or multiple processes
>> 2 - live debug session tracing for being able to see how you crashed where 
>> trace data is for current process only
>> 
>> For post mortem tracing, the trace top level command seemed to make sense 
>> here because there are no other target commands that act on more than one 
>> target. So "trace load" makes sense to me here for loading one or more 
>> traces. The idea is the trace JSON file has enough info to completely load 
>> up the state of the trace so we can symbolicate, dump and step around in 
>> history. So I would vote to keep "trace load" at the very least because it 
>> can create one or more targets. Options can be added to display the 
>> processes if needed:
>> 
>> (lldb) trace list 
>> 
>> But we could move "trace dump" over into "target trace dump" or "process 
>> trace dump" since that is effectively how we are coding these patches.
>> 
>> For live debugging where we gather trace data through the process plug-in, 
>> we will have a live process that may or may not have trace data. If tracing 
>> isn't available we will not be able to dump anything. But I would like to 
>> see process/thread commands for this scenario:
>> 
>> - process trace start/stop (only succeeds if we can gather trace data 
>> through the process plug-in)
>> - thread trace start/stop (which can succeed only if current tracing can 
>> enable tracing for only one thread)
>> 
>> Not sure if we need "process trace save" or "thread trace save" as the 
>> saving can be done as an option to "process trace stop --save /path/to/save"
>> 
>> So I am all for fitting these commands in where they need to go.
>> 
>> > 
>> > On 21/09/2020 22:17, Walter via lldb-dev wrote:
>> >> Thanks for your feedback Fangrui, I've just been checking Capn' Proto
>> >> and it looks really good. I'll keep it in mind in the design and see how
>> >> it can optimize the overall data transfer.
>> > 
>> > I'm not sure how Cap'n Proto comes into play here. The way I 

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Greg Clayton via lldb-dev]

We spoke a bit after Panel's comments which made sense and we propose the 
commands Walter sent below. Let us know what everyone thinks of this 
organization of the command structure!

> On Oct 1, 2020, at 1:32 PM, Walter  wrote:
> 
> After a chat with Greg, we agreed on this set of commands
> 
> 
> trace load /path/to/json
> 
> process trace start/stop
> process trace save /path/to/json
> 
> thread trace start/stop
> thread trace dump [instructions | functions]
> 
> 
> Il giorno gio 1 ott 2020 alle ore 13:21 Greg Clayton  > ha scritto:
> 
> 
> > On Oct 1, 2020, at 7:08 AM, Pavel Labath via lldb-dev 
> > mailto:lldb-dev@lists.llvm.org>> wrote:
> > 
> > Thank you for writing this Walter. I think this document will be a
> > useful reference both now and in the future.
> > 
> > The part that's not clear to me is what is the story with multi-process
> > traces. The file format enables those, but it's not clear how are they
> > going be created or used. Can you elaborate more on what you intend to
> > use those for?
> 
> Mainly for system trace kinds of things where an entire system gets traced.
> 
> > 
> > The main reason I am asking that is because I am thinking about the
> > proposed command structure. I'm wondering if it would not be better to
> > fit this into the existing target/process/thread commands instead of
> > adding a new top-level command. For example, one could imagine the
> > following set of commands:
> > 
> > - "process trace start" + "thread trace start" instead of "thread trace
> > [tid]". That would be similar to "process continue" + "thread continue".
> > - "thread trace dump [tid]" instead of "trace dump [-t tid]". That would
> > be similar to "thread continue" and other thread control commands.
> > - "target create --trace" instead of "trace load". (analogous to target
> > create --core).
> > - "process trace save" instead of "trace save" -- (mostly) analogous to
> > "process save-core"
> 
> > I am thinking this composition may fit in better into the existing lldb
> > command landscape, though I also see the appeal in grouping everything
> > trace-related under a single top-level command. What do you think?
> > 
> > The main place where this idea breaks down is the multi-process traces.
> > While we could certainly make "target create --trace" create multiple
> > targets, that would be fairly unusual. OTOH, the whole concept of having
> > multiple targets share something is a pretty unusual thing for lldb.
> > That's why I'd like to hear more about where you want to go with this idea.
> 
> I kind of see tracing has having two sides:
> 1 - post mortem tracing for individual or multiple processes
> 2 - live debug session tracing for being able to see how you crashed where 
> trace data is for current process only
> 
> For post mortem tracing, the trace top level command seemed to make sense 
> here because there are no other target commands that act on more than one 
> target. So "trace load" makes sense to me here for loading one or more 
> traces. The idea is the trace JSON file has enough info to completely load up 
> the state of the trace so we can symbolicate, dump and step around in 
> history. So I would vote to keep "trace load" at the very least because it 
> can create one or more targets. Options can be added to display the processes 
> if needed:
> 
> (lldb) trace list 
> 
> But we could move "trace dump" over into "target trace dump" or "process 
> trace dump" since that is effectively how we are coding these patches.
> 
> For live debugging where we gather trace data through the process plug-in, we 
> will have a live process that may or may not have trace data. If tracing 
> isn't available we will not be able to dump anything. But I would like to see 
> process/thread commands for this scenario:
> 
> - process trace start/stop (only succeeds if we can gather trace data through 
> the process plug-in)
> - thread trace start/stop (which can succeed only if current tracing can 
> enable tracing for only one thread)
> 
> Not sure if we need "process trace save" or "thread trace save" as the saving 
> can be done as an option to "process trace stop --save /path/to/save"
> 
> So I am all for fitting these commands in where they need to go.
> 
> > 
> > On 21/09/2020 22:17, Walter via lldb-dev wrote:
> >> Thanks for your feedback Fangrui, I've just been checking Capn' Proto
> >> and it looks really good. I'll keep it in mind in the design and see how
> >> it can optimize the overall data transfer.
> > 
> > I'm not sure how Cap'n Proto comes into play here. The way I understand
> > it, the real data is contained in a separate file in the specialized
> > intel format and the json is just for the metadata. I'd expect the
> > metadata file to be small even for enormous traces, so I'm not sure
> > what's to be gained by optimizing it.
> > 
> > pl
> > 
> > ___
> > lldb-dev mailing list
> > lldb-dev@lists.llvm.org 

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Walter via lldb-dev]

After a chat with Greg, we agreed on this set of commands


trace load /path/to/json process trace start/stop process trace save
/path/to/json thread trace start/stop thread trace dump [instructions |
functions]

Il giorno gio 1 ott 2020 alle ore 13:21 Greg Clayton 
ha scritto:

>
>
> > On Oct 1, 2020, at 7:08 AM, Pavel Labath via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> >
> > Thank you for writing this Walter. I think this document will be a
> > useful reference both now and in the future.
> >
> > The part that's not clear to me is what is the story with multi-process
> > traces. The file format enables those, but it's not clear how are they
> > going be created or used. Can you elaborate more on what you intend to
> > use those for?
>
> Mainly for system trace kinds of things where an entire system gets traced.
>
> >
> > The main reason I am asking that is because I am thinking about the
> > proposed command structure. I'm wondering if it would not be better to
> > fit this into the existing target/process/thread commands instead of
> > adding a new top-level command. For example, one could imagine the
> > following set of commands:
> >
> > - "process trace start" + "thread trace start" instead of "thread trace
> > [tid]". That would be similar to "process continue" + "thread continue".
> > - "thread trace dump [tid]" instead of "trace dump [-t tid]". That would
> > be similar to "thread continue" and other thread control commands.
> > - "target create --trace" instead of "trace load". (analogous to target
> > create --core).
> > - "process trace save" instead of "trace save" -- (mostly) analogous to
> > "process save-core"
>
> > I am thinking this composition may fit in better into the existing lldb
> > command landscape, though I also see the appeal in grouping everything
> > trace-related under a single top-level command. What do you think?
> >
> > The main place where this idea breaks down is the multi-process traces.
> > While we could certainly make "target create --trace" create multiple
> > targets, that would be fairly unusual. OTOH, the whole concept of having
> > multiple targets share something is a pretty unusual thing for lldb.
> > That's why I'd like to hear more about where you want to go with this
> idea.
>
> I kind of see tracing has having two sides:
> 1 - post mortem tracing for individual or multiple processes
> 2 - live debug session tracing for being able to see how you crashed where
> trace data is for current process only
>
> For post mortem tracing, the trace top level command seemed to make sense
> here because there are no other target commands that act on more than one
> target. So "trace load" makes sense to me here for loading one or more
> traces. The idea is the trace JSON file has enough info to completely load
> up the state of the trace so we can symbolicate, dump and step around in
> history. So I would vote to keep "trace load" at the very least because it
> can create one or more targets. Options can be added to display the
> processes if needed:
>
> (lldb) trace list 
>
> But we could move "trace dump" over into "target trace dump" or "process
> trace dump" since that is effectively how we are coding these patches.
>
> For live debugging where we gather trace data through the process plug-in,
> we will have a live process that may or may not have trace data. If tracing
> isn't available we will not be able to dump anything. But I would like to
> see process/thread commands for this scenario:
>
> - process trace start/stop (only succeeds if we can gather trace data
> through the process plug-in)
> - thread trace start/stop (which can succeed only if current tracing can
> enable tracing for only one thread)
>
> Not sure if we need "process trace save" or "thread trace save" as the
> saving can be done as an option to "process trace stop --save /path/to/save"
>
> So I am all for fitting these commands in where they need to go.
>
> >
> > On 21/09/2020 22:17, Walter via lldb-dev wrote:
> >> Thanks for your feedback Fangrui, I've just been checking Capn' Proto
> >> and it looks really good. I'll keep it in mind in the design and see how
> >> it can optimize the overall data transfer.
> >
> > I'm not sure how Cap'n Proto comes into play here. The way I understand
> > it, the real data is contained in a separate file in the specialized
> > intel format and the json is just for the metadata. I'd expect the
> > metadata file to be small even for enormous traces, so I'm not sure
> > what's to be gained by optimizing it.
> >
> > pl
> >
> > ___
> > lldb-dev mailing list
> > lldb-dev@lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>

-- 
- Walter Erquínigo Pezo
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Greg Clayton via lldb-dev]

> On Oct 1, 2020, at 7:08 AM, Pavel Labath via lldb-dev 
>  wrote:
> 
> Thank you for writing this Walter. I think this document will be a
> useful reference both now and in the future.
> 
> The part that's not clear to me is what is the story with multi-process
> traces. The file format enables those, but it's not clear how are they
> going be created or used. Can you elaborate more on what you intend to
> use those for?

Mainly for system trace kinds of things where an entire system gets traced.

> 
> The main reason I am asking that is because I am thinking about the
> proposed command structure. I'm wondering if it would not be better to
> fit this into the existing target/process/thread commands instead of
> adding a new top-level command. For example, one could imagine the
> following set of commands:
> 
> - "process trace start" + "thread trace start" instead of "thread trace
> [tid]". That would be similar to "process continue" + "thread continue".
> - "thread trace dump [tid]" instead of "trace dump [-t tid]". That would
> be similar to "thread continue" and other thread control commands.
> - "target create --trace" instead of "trace load". (analogous to target
> create --core).
> - "process trace save" instead of "trace save" -- (mostly) analogous to
> "process save-core"

> I am thinking this composition may fit in better into the existing lldb
> command landscape, though I also see the appeal in grouping everything
> trace-related under a single top-level command. What do you think?
> 
> The main place where this idea breaks down is the multi-process traces.
> While we could certainly make "target create --trace" create multiple
> targets, that would be fairly unusual. OTOH, the whole concept of having
> multiple targets share something is a pretty unusual thing for lldb.
> That's why I'd like to hear more about where you want to go with this idea.

I kind of see tracing has having two sides:
1 - post mortem tracing for individual or multiple processes
2 - live debug session tracing for being able to see how you crashed where 
trace data is for current process only

For post mortem tracing, the trace top level command seemed to make sense here 
because there are no other target commands that act on more than one target. So 
"trace load" makes sense to me here for loading one or more traces. The idea is 
the trace JSON file has enough info to completely load up the state of the 
trace so we can symbolicate, dump and step around in history. So I would vote 
to keep "trace load" at the very least because it can create one or more 
targets. Options can be added to display the processes if needed:

(lldb) trace list 

But we could move "trace dump" over into "target trace dump" or "process trace 
dump" since that is effectively how we are coding these patches.

For live debugging where we gather trace data through the process plug-in, we 
will have a live process that may or may not have trace data. If tracing isn't 
available we will not be able to dump anything. But I would like to see 
process/thread commands for this scenario:

- process trace start/stop (only succeeds if we can gather trace data through 
the process plug-in)
- thread trace start/stop (which can succeed only if current tracing can enable 
tracing for only one thread)

Not sure if we need "process trace save" or "thread trace save" as the saving 
can be done as an option to "process trace stop --save /path/to/save"

So I am all for fitting these commands in where they need to go.

> 
> On 21/09/2020 22:17, Walter via lldb-dev wrote:
>> Thanks for your feedback Fangrui, I've just been checking Capn' Proto
>> and it looks really good. I'll keep it in mind in the design and see how
>> it can optimize the overall data transfer.
> 
> I'm not sure how Cap'n Proto comes into play here. The way I understand
> it, the real data is contained in a separate file in the specialized
> intel format and the json is just for the metadata. I'd expect the
> metadata file to be small even for enormous traces, so I'm not sure
> what's to be gained by optimizing it.
> 
> pl
> 
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Walter via lldb-dev]

Hi Pavel, thanks for the comments. I'll reply inline

> The part that's not clear to me is what is the story with multi-process
traces. The file format enables those, but it's not clear how are they
going be created or used. Can you elaborate more on what you intend to
use those for?

Something we are doing at Facebook is having a global Intel PT collector
that can trace all processes of a given machine for some seconds. This can
produce a multi-process trace. I imagine these traces won't ever be
generated by LLDB, though. Having one single json trace file for this is
going to be useful for sharing the trace more easily. Multi-process tracing
is also something you can do with the perf tool, so It's not uncommon.

There are some technical details that are worth mentioning as well. Intel
PT offers two main modes of tracing: single thread tracing and logical CPU
tracing.
- The first one is the easiest to implement, but it requires having a
dedicated buffer per thread, which can consume too much RAM if there are
thousands of threads traced. It also adds a little bit of performance cost,
as the kernel disables and enables tracing whenever there's a context
switch.
- The other mode, logical CPU tracing, traces all the activity in one
logical core and uses one single buffer. Also it is more performant as the
kernel doesn't disable tracing intermittently. Sadly, that trace contains
no information regarding context switches, so a separated context switch
trace is used for splitting this big trace into per-thread subtraces. The
decoder we are implementing eventually will be able to do this splitting,
and it will require being fed with the information of all processes. This
is also a reason why allowing multi-process traces is important.

Regarding the commands structure, I'd prefer to keep it under "trace" for
now, because of the multi-process case and because we still have no users
that can report feedback. Very soon we'll start building some tools around
this feature, so we'll have more concrete experiences to share. Then it'll
be good to sync up and revisit the structure.

Btw, the gdb implementation of this kind of tracing is under the "record"
main command (
https://sourceware.org/gdb/current/onlinedocs/gdb/Process-Record-and-Replay.html).
I think this allows for some flexibility, as each trace plugin has
different characteristics.

> I'm not sure how Cap'n Proto comes into play here. The way I understand
it, the real data is contained in a separate file in the specialized intel
format and the json is just for the metadata. I'd expect the metadata file
to be small even for enormous traces, so I'm not sure what's to be gained
by optimizing it.

I didn't mention it in that email, but there is some additional information
that we'll eventually include in the traces, like the context-switch trace
I mentioned above. I think that we could probably use Cap'n Proto for cases
like this. We might also not use it at all as well, but it was nice to
learn about it and keep it in mind.


Thanks,
Walter

Il giorno gio 1 ott 2020 alle ore 07:08 Pavel Labath  ha
scritto:

> Thank you for writing this Walter. I think this document will be a
> useful reference both now and in the future.
>
> The part that's not clear to me is what is the story with multi-process
> traces. The file format enables those, but it's not clear how are they
> going be created or used. Can you elaborate more on what you intend to
> use those for?
>
> The main reason I am asking that is because I am thinking about the
> proposed command structure. I'm wondering if it would not be better to
> fit this into the existing target/process/thread commands instead of
> adding a new top-level command. For example, one could imagine the
> following set of commands:
>
> - "process trace start" + "thread trace start" instead of "thread trace
> [tid]". That would be similar to "process continue" + "thread continue".
> - "thread trace dump [tid]" instead of "trace dump [-t tid]". That would
> be similar to "thread continue" and other thread control commands.
> - "target create --trace" instead of "trace load". (analogous to target
> create --core).
> - "process trace save" instead of "trace save" -- (mostly) analogous to
> "process save-core"
>
> I am thinking this composition may fit in better into the existing lldb
> command landscape, though I also see the appeal in grouping everything
> trace-related under a single top-level command. What do you think?
>
> The main place where this idea breaks down is the multi-process traces.
> While we could certainly make "target create --trace" create multiple
> targets, that would be fairly unusual. OTOH, the whole concept of having
> multiple targets share something is a pretty unusual thing for lldb.
> That's why I'd like to hear more about where you want to go with this idea.
>
>
> On 21/09/2020 22:17, Walter via lldb-dev wrote:
> > Thanks for your feedback Fangrui, I've just been checking Capn' Proto
> > and it looks 

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Pavel Labath via lldb-dev]

Thank you for writing this Walter. I think this document will be a
useful reference both now and in the future.

The part that's not clear to me is what is the story with multi-process
traces. The file format enables those, but it's not clear how are they
going be created or used. Can you elaborate more on what you intend to
use those for?

The main reason I am asking that is because I am thinking about the
proposed command structure. I'm wondering if it would not be better to
fit this into the existing target/process/thread commands instead of
adding a new top-level command. For example, one could imagine the
following set of commands:

- "process trace start" + "thread trace start" instead of "thread trace
[tid]". That would be similar to "process continue" + "thread continue".
- "thread trace dump [tid]" instead of "trace dump [-t tid]". That would
be similar to "thread continue" and other thread control commands.
- "target create --trace" instead of "trace load". (analogous to target
create --core).
- "process trace save" instead of "trace save" -- (mostly) analogous to
"process save-core"

I am thinking this composition may fit in better into the existing lldb
command landscape, though I also see the appeal in grouping everything
trace-related under a single top-level command. What do you think?

The main place where this idea breaks down is the multi-process traces.
While we could certainly make "target create --trace" create multiple
targets, that would be fairly unusual. OTOH, the whole concept of having
multiple targets share something is a pretty unusual thing for lldb.
That's why I'd like to hear more about where you want to go with this idea.


On 21/09/2020 22:17, Walter via lldb-dev wrote:
> Thanks for your feedback Fangrui, I've just been checking Capn' Proto
> and it looks really good. I'll keep it in mind in the design and see how
> it can optimize the overall data transfer.

I'm not sure how Cap'n Proto comes into play here. The way I understand
it, the real data is contained in a separate file in the specialized
intel format and the json is just for the metadata. I'd expect the
metadata file to be small even for enormous traces, so I'm not sure
what's to be gained by optimizing it.

pl

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Walter via lldb-dev]

Thanks for your feedback Fangrui, I've just been checking Capn' Proto and
it looks really good. I'll keep it in mind in the design and see how it can
optimize the overall data transfer.

- Walter

Il giorno dom 20 set 2020 alle ore 11:24 Fangrui Song 
ha scritto:

> On 2020-09-18, Eric Christopher via lldb-dev wrote:
> >Hi Walter,
> >
> >I've only done a brief scan of the document but, in general, I'm favorable
> >of the goals, aim, and approach. Something I think would be good would be
> >to compare/contrast against rr as an "exploring alternatives" section of
> >the document. I think the document should also be made available/adapted
> to
> >be part of the documentation on "why lldb is implementing this
> feature/what
> >it can be used for/why".
> >
> >Thanks so much for starting this and looking forward to the work and
> >collaboration.
> >
> >-eric
>
> Same. I am really excited that this work will open up possibilities for
> reverse debugging, which is the most important factor impeding me from
> migrating (from gdb) to lldb :)
>
> For unit tests, a json format tracing record is probably convenient, but
> for practical usage we may need a compacter format, e.g. Cap'n Proto
> used by rr
> (https://robert.ocallahan.org/2017/08/stabilizing-rr-trace-format.html)
> Hope the framework can be easily adapted to such a compact format.
>
> >On Thu, Sep 17, 2020 at 8:28 PM Walter via lldb-dev <
> lldb-dev@lists.llvm.org>
> >wrote:
> >
> >> Hi all,
> >>
> >>
> >>
> >> Here I propose, along with Greg Clayton, Processor Trace support for
> LLDB. I’m attaching a link to the document that contains this proposal if
> that’s easier to read for you:
> https://docs.google.com/document/d/1cOVTGp1sL_HBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI/edit#heading=h.t5mblb9ugv8f
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1cOVTGp1sL-5FHBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI_edit-23heading-3Dh.t5mblb9ugv8f=DwMGaQ=5VD0RTtNlTh3ycd41b3MUw=erxV6KMIZvIQjyWYW8YpOiKz-WqJt4giKQA34YMHsRY=DuuwXHUQJpW4TcCay4hPsBund-eBI2uVaVimqEPsp5k=o6vqoYYbn-Tz_d34hoLJvWhEnnhracOO6yDsMzq8wR0=>.
> Please make any comments in this mail list.
> >>
> >>
> >>
> >> If you want to quickly know what Processor Trace can do, you can read
> this https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__easyperf.net_blog_2019_08_23_Intel-2DProcessor-2DTrace=DwMGaQ=5VD0RTtNlTh3ycd41b3MUw=erxV6KMIZvIQjyWYW8YpOiKz-WqJt4giKQA34YMHsRY=DuuwXHUQJpW4TcCay4hPsBund-eBI2uVaVimqEPsp5k=iaErHaf8byXlZb1YFUk0BpQ-duMhNouUUMyktLm3soQ=
> >.
> >>
> >>
> >>
> >> Any comments are appreciated, especially the ones regarding the
> commands the user will interact with.
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Walter Erquinigo.
> >>
> >>
> >>
> >>
> >>
> >> # RFC: Processor Trace Support in LLDB
> >>
> >>
> >>
> >>
> >>
> >> # What is processor tracing?
> >>
> >>
> >>
> >> Processor tracing works by capturing information about the execution of
> a process so that the control flow of the program can be reconstructed
> later. Implementations of this are Intel Processor Trace for X86, x86_64 ([
> https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html))
> and ARM CoreSight for some ARM devices ([
> https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace](https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace)
> ).
> >>
> >>
> >>
> >> As a clarifying example, with these technologies it’s possible to trace
> all the threads of a process, and after the process has finished,
> reconstruct every single instruction address each thread has executed. This
> could include some additional information like timestamps, async CPU
> events, kernel instructions, bus clock ratio changes, etc. On the other
> hand, memory and registers are not traced as a way to limit the size of the
> trace.
> >>
> >>
> >>
> >>
> >>
> >> # Intel Processor Trace as the first implementation
> >>
> >>
> >>
> >> We’ll focus on Intel Processor Trace (Intel PT), but in a generic way
> so that in the future similar technologies can be onboarded in LLDB.
> >>
> >>
> >>
> >> Intel PT has the following features:
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> *   Control flow tracing in a highly encoded format
> >>
> >> *   3% to 5% slowdown when capturing
> >>
> >> *   No memory nor registers captured
> >>
> >> *   Kernel tracing support
> >>
> >> *   Timestamps of branches are produced, which can be used for profiling
> >>
> >> *   Adjustable size of trace buffer
> >>
> >> *   Supported on most Intel CPUs since 2015
> >>
> >> *   X86 and x86_64 only
> >>
> >> *   Official support only on Linux
> >>
> >> *   Basic support on Windows
> >>
> >> *   Decoding/analysis can be done on any operating system
> >>
> >>
> >>
> >> A very nice introduction to Intel PT can be found [
> 

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Walter via lldb-dev]

Hi Eric, thanks for the feedback
> Something I think would be good would be to compare/contrast against rr
as an "exploring alternatives" section of the document.

I'll include that. I've done some comparative research on rr and I think I
can provide valuable input.

> I think the document should also be made available/adapted to be part of
the documentation on "why lldb is implementing this feature/what it can be
used for/why".

I think this information is scattered throughout the document, but I'll
make sure to answer this in one of the first paragraphs.

Thanks!
- Walter

Il giorno ven 18 set 2020 alle ore 19:58 Eric Christopher <
echri...@gmail.com> ha scritto:

> Hi Walter,
>
> I've only done a brief scan of the document but, in general, I'm favorable
> of the goals, aim, and approach. Something I think would be good would be
> to compare/contrast against rr as an "exploring alternatives" section of
> the document. I think the document should also be made available/adapted to
> be part of the documentation on "why lldb is implementing this feature/what
> it can be used for/why".
>
> Thanks so much for starting this and looking forward to the work and
> collaboration.
>
> -eric
>
> On Thu, Sep 17, 2020 at 8:28 PM Walter via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Hi all,
>>
>>
>>
>> Here I propose, along with Greg Clayton, Processor Trace support for LLDB. 
>> I’m attaching a link to the document that contains this proposal if that’s 
>> easier to read for you: 
>> https://docs.google.com/document/d/1cOVTGp1sL_HBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI/edit#heading=h.t5mblb9ugv8f
>>  
>> .
>>  Please make any comments in this mail list.
>>
>>
>>
>> If you want to quickly know what Processor Trace can do, you can read this 
>> https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace 
>> .
>>
>>
>>
>> Any comments are appreciated, especially the ones regarding the commands the 
>> user will interact with.
>>
>>
>>
>> Thanks,
>>
>> Walter Erquinigo.
>>
>>
>>
>>
>>
>> # RFC: Processor Trace Support in LLDB
>>
>>
>>
>>
>>
>> # What is processor tracing?
>>
>>
>>
>> Processor tracing works by capturing information about the execution of a 
>> process so that the control flow of the program can be reconstructed later. 
>> Implementations of this are Intel Processor Trace for X86, x86_64 
>> ([https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html))
>>  and ARM CoreSight for some ARM devices 
>> ([https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace](https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace)).
>>
>>
>>
>> As a clarifying example, with these technologies it’s possible to trace all 
>> the threads of a process, and after the process has finished, reconstruct 
>> every single instruction address each thread has executed. This could 
>> include some additional information like timestamps, async CPU events, 
>> kernel instructions, bus clock ratio changes, etc. On the other hand, memory 
>> and registers are not traced as a way to limit the size of the trace.
>>
>>
>>
>>
>>
>> # Intel Processor Trace as the first implementation
>>
>>
>>
>> We’ll focus on Intel Processor Trace (Intel PT), but in a generic way so 
>> that in the future similar technologies can be onboarded in LLDB.
>>
>>
>>
>> Intel PT has the following features:
>>
>>
>>
>>
>>
>>
>>
>> *   Control flow tracing in a highly encoded format
>>
>> *   3% to 5% slowdown when capturing
>>
>> *   No memory nor registers captured
>>
>> *   Kernel tracing support
>>
>> *   Timestamps of branches are produced, which can be used for profiling
>>
>> *   Adjustable size of trace buffer
>>
>> *   Supported on most Intel CPUs since 2015
>>
>> *   X86 and x86_64 only
>>
>> *   Official support only on Linux
>>
>> *   Basic support on Windows
>>
>> *   Decoding/analysis can be done on any operating system
>>
>>
>>
>> A very nice introduction to Intel PT can be found 
>> [https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html)
>>  and 
>> [https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace](https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace).
>>  Totally recommended to fully grasp the impact of 

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Fangrui Song via lldb-dev]

On 2020-09-18, Eric Christopher via lldb-dev wrote:

Hi Walter,

I've only done a brief scan of the document but, in general, I'm favorable
of the goals, aim, and approach. Something I think would be good would be
to compare/contrast against rr as an "exploring alternatives" section of
the document. I think the document should also be made available/adapted to
be part of the documentation on "why lldb is implementing this feature/what
it can be used for/why".

Thanks so much for starting this and looking forward to the work and
collaboration.

-eric


Same. I am really excited that this work will open up possibilities for
reverse debugging, which is the most important factor impeding me from
migrating (from gdb) to lldb :)

For unit tests, a json format tracing record is probably convenient, but
for practical usage we may need a compacter format, e.g. Cap'n Proto
used by rr
(https://robert.ocallahan.org/2017/08/stabilizing-rr-trace-format.html)
Hope the framework can be easily adapted to such a compact format.


On Thu, Sep 17, 2020 at 8:28 PM Walter via lldb-dev 
wrote:


Hi all,



Here I propose, along with Greg Clayton, Processor Trace support for LLDB. I’m attaching a link to the 
document that contains this proposal if that’s easier to read for you: 
https://docs.google.com/document/d/1cOVTGp1sL_HBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI/edit#heading=h.t5mblb9ugv8f 
.
 Please make any comments in this mail list.



If you want to quickly know what Processor Trace can do, you can read this 
https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace 
.



Any comments are appreciated, especially the ones regarding the commands the 
user will interact with.



Thanks,

Walter Erquinigo.





# RFC: Processor Trace Support in LLDB





# What is processor tracing?



Processor tracing works by capturing information about the execution of a 
process so that the control flow of the program can be reconstructed later. 
Implementations of this are Intel Processor Trace for X86, x86_64 
([https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html))
 and ARM CoreSight for some ARM devices 
([https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace](https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace)).



As a clarifying example, with these technologies it’s possible to trace all the 
threads of a process, and after the process has finished, reconstruct every 
single instruction address each thread has executed. This could include some 
additional information like timestamps, async CPU events, kernel instructions, 
bus clock ratio changes, etc. On the other hand, memory and registers are not 
traced as a way to limit the size of the trace.





# Intel Processor Trace as the first implementation



We’ll focus on Intel Processor Trace (Intel PT), but in a generic way so that 
in the future similar technologies can be onboarded in LLDB.



Intel PT has the following features:







*   Control flow tracing in a highly encoded format

*   3% to 5% slowdown when capturing

*   No memory nor registers captured

*   Kernel tracing support

*   Timestamps of branches are produced, which can be used for profiling

*   Adjustable size of trace buffer

*   Supported on most Intel CPUs since 2015

*   X86 and x86_64 only

*   Official support only on Linux

*   Basic support on Windows

*   Decoding/analysis can be done on any operating system



A very nice introduction to Intel PT can be found 
[https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html)
 and 
[https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace](https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace).
 Totally recommended to fully grasp the impact of this project.



More technical details are in 
[https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt](https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt).



Even more technical details are in the processor manual 

Re: [lldb-dev] RFC: Processor Trace Support in LLDB

[Vedant Kumar via lldb-dev]

Hi Walter & Greg,

Thanks for sharing this RFC, and for your work in this area.

> On Sep 17, 2020, at 5:28 PM, Walter via lldb-dev  
> wrote:
> 
> Hi all,
>  
> Here I propose, along with Greg Clayton, Processor Trace support for LLDB. 
> I’m attaching a link to the document that contains this proposal if that’s 
> easier to read for you: 
> https://docs.google.com/document/d/1cOVTGp1sL_HBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI/edit#heading=h.t5mblb9ugv8f
>  
> .
>  Please make any comments in this mail list.
>  
>  
> If you want to quickly know what Processor Trace can do, you can read this 
> https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace 
> .
>  
> Any comments are appreciated, especially the ones regarding the commands the 
> user will interact with. 
>  
> Thanks,
> Walter Erquinigo.
>  
>  
> # RFC: Processor Trace Support in LLDB
>  
>  
> # What is processor tracing?
>  
> Processor tracing works by capturing information about the execution of a 
> process so that the control flow of the program can be reconstructed later. 
> Implementations of this are Intel Processor Trace for X86, x86_64 
> ([https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html)
>  
> )
>  and ARM CoreSight for some ARM devices 
> ([https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace](https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace)
>  
> ).
>  
>  
> As a clarifying example, with these technologies it’s possible to trace all 
> the threads of a process, and after the process has finished, reconstruct 
> every single instruction address each thread has executed. This could include 
> some additional information like timestamps, async CPU events, kernel 
> instructions, bus clock ratio changes, etc. On the other hand, memory and 
> registers are not traced as a way to limit the size of the trace.
>  
>  
> # Intel Processor Trace as the first implementation
>  
> We’ll focus on Intel Processor Trace (Intel PT), but in a generic way so that 
> in the future similar technologies can be onboarded in LLDB.
>  
> Intel PT has the following features:
>  
>  
>  
> *   Control flow tracing in a highly encoded format
> *   3% to 5% slowdown when capturing
> *   No memory nor registers captured
> *   Kernel tracing support
> *   Timestamps of branches are produced, which can be used for profiling
> *   Adjustable size of trace buffer
> *   Supported on most Intel CPUs since 2015
> *   X86 and x86_64 only
> *   Official support only on Linux
> *   Basic support on Windows
> *   Decoding/analysis can be done on any operating system
>  
> A very nice introduction to Intel PT can be found 
> [https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html)
>  
> 
>  and 
> [https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace](https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace)
>  
> .
>  Totally recommended to fully grasp the impact of this project. 
>  
> More technical details are in 
> [https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt](https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt)
>  
> .
>  
>  
> Even more technical details are in the processor manual 
>