Re: [lttng-dev] porting LTTng to Go(lang)?

[Mathieu Desnoyers]

- On Jun 3, 2017, at 3:45 PM, Philippe Proulx  
wrote: 

> On Fri, Jun 2, 2017 at 10:24 PM, craig harmer < [ mailto:charms...@gmail.com |
> charms...@gmail.com ] > wrote:

>> hi all,

>> the suggestion to write a pre-processor for LTTng tracepoints is interesting 
>> and
>> may be a good idea. however, i'm not sure that it will provide a general
>> purpose solution.

>> the problem i for see is that tracepoint invocations, at least, should be
>> written in the syntax of the language they're being used in (a programmer 
>> would
>> would expect the tracepoint() function call to look like any other function
>> call in the language).

>> so the tracepoint parser needs to understand the function invocation syntax 
>> of
>> each language. it also needs to understand at least some of the rest of the
>> syntax of the language -- in the case of Go it at least needs to be able
>> recognize "import" statements and import those files and it needs to 
>> understand
>> when something that looks like a tracepoint() is actually embedded in a 
>> comment
>> or string. it also needs to emit code that matches the syntax of the 
>> language.

>> so a general purpose lttng tracepoint parsing tool would need at least a 
>> limited
>> understanding of the syntax and grammar of each language for reading the 
>> source
>> file in and then writing the source file out. there could be code in the 
>> middle
>> that's common across languages, but probably not a lot. perhaps an lttng
>> tracepoint parsing tool could be better structured as a common framework for
>> language specific tracepoint parsers to be plugged in to.

>> but writing a parser that actually understands the grammar of a language is 
>> not
>> easy. if i was really going to do this, i'd be tempted to see if i could just
>> take this from gcc (which has front-ends that supports C, C++, Go, and other
>> languages (i think)).

>> and then there's the tracepoint definitions. they could be written using the
>> barectf() tool that Philippe mentions, but i'm not sure if that would be
>> satisfactory for most programmers, who would prefer to write tracepoint
>> definitions in the same language that they are programming in. they would 
>> also
>> need to support the data types of the language (i'm thinking Python here as 
>> an
>> example, where tracepoints would be expected to support most of the Python
>> basic data types).

> There's a little misunderstanding here.

> The idea is to have a universal tracepoint definition language (and what I
> suggested is that a subset of CTF 2, of which the syntax is JSON (or YAML 
> which
> translates to JSON directly), could be used for this) which translates to
> functions, classes, macros, etc. in selected programming languages. Depending
> on the destination language, we can generate serialization functions directly
> or wrappers which use LTTng-UST behind the scenes. So in your case, this tool
> would generate Go source files which would contain public functions which
> correspond to tracepoints. We never considered parsing an existing programming
> language to infer tracepoints.

> Of course this requires the developer to define tracepoints in this universal
> language. That's an additional developing step needed to allow the developer 
> to
> use tracepoints in his/her favorite language. But there are other advantages,
> like generating tracepoints in two different languages from the same 
> tracepoint
> definitions. Like Mathieu wrote, it's the same spirit as Flex and Bison. With
> Bison, you don't use the C preprocessor to write your grammar rules: the tool
> reads a Bison input file and generates a parser source file. Just like barectf
> reads a YAML configuration file which defines stream and event classes and
> generates tracer source files.

The problem is that this is often not enough, and we sometimes need to have 
pre/post code run before/after the serialization to prepare the arguments. 
This happens for instance in lttng-modules with the 
LTTNG_TRACEPOINT_EVENT_CODE() 
macro. 

Ideally we should find a way to glue together this high-level description 
and language-native helper code. 

Thoughts ? 

Thanks, 

Mathieu 

> Phil

>> because Go seems to be similar to C i'm strongly tempted to just co-opt the C
>> pre-processor and existing LTTng tracepoint machinery and modify the LTTng
>> header files to generate valid Go code for a tracepoint definition. this does
>> add a LTTng tracepoint tool to the steps required to compile Go, except that
>> the tracepoint tool is called "cpp" (plus header files).

>> i don't really know Go yet and i hae not been assigned to get LTTng to work 
>> in
>> Go, so this is all quite preliminary. but i think the project i'm about to
>> start on really needs LTTng or LTTng-like tracepoint functionality so i'm
>> motivated to see if this can work.

>> i'm going to investigate some more.

>> --craig

>> On Fri, Jun 2, 2017 at 10:05 AM, Philippe Proulx < [
>> 

Re: [lttng-dev] porting LTTng to Go(lang)?

[Philippe Proulx]

On Fri, Jun 2, 2017 at 10:24 PM, craig harmer  wrote:

> hi all,
>
> the suggestion to write a pre-processor for LTTng tracepoints is
> interesting and may be a good idea.  however, i'm not sure that it will
> provide a general purpose solution.
>
> the problem i for see is that tracepoint invocations, at least, should be
> written in the syntax of the language they're being used in (a programmer
> would would expect the tracepoint() function call to look like any other
> function call in the language).
>
> so the tracepoint parser needs to understand the function invocation
> syntax of each language.  it also needs to understand at least some of the
> rest of the syntax of the language -- in the case of Go it at least needs
> to be able recognize "import" statements and import those files and it
> needs to understand when something that looks like a tracepoint() is
> actually embedded in a comment or string.  it also needs to emit code that
> matches the syntax of the language.
>
> so a general purpose lttng tracepoint parsing tool would need at least a
> limited understanding of the syntax and grammar of each language for
> reading the source file in and then writing the source file out.  there
> could be code in the middle that's common across languages, but probably
> not a lot.  perhaps an lttng tracepoint parsing tool could be better
> structured as a common framework for language specific tracepoint parsers
> to be plugged in to.
>
> but writing a parser that actually understands the grammar of a language
> is not easy.  if i was really going to do this, i'd be tempted to see if i
> could just take this from gcc (which has front-ends that supports C, C++,
> Go, and other languages (i think)).
>
> and then there's the tracepoint definitions.  they could be written using
> the barectf() tool that Philippe mentions, but i'm not sure if that would
> be satisfactory for most programmers, who would prefer to write tracepoint
> definitions in the same language that they are programming in.  they would
> also need to support the data types of the language (i'm thinking Python
> here as an example, where tracepoints would be expected to support most of
> the Python basic data types).
>

There's a little misunderstanding here.

The idea is to have a universal tracepoint definition language (and what I
suggested is that a subset of CTF 2, of which the syntax is JSON (or YAML
which translates to JSON directly), could be used for this) which
translates to functions, classes, macros, etc. in selected programming
languages. Depending on the destination language, we can generate
serialization functions directly or wrappers which use LTTng-UST behind the
scenes. So in your case, this tool would generate Go source files which
would contain public functions which correspond to tracepoints. We never
considered parsing an existing programming language to infer tracepoints.

Of course this requires the developer to define tracepoints in this
universal language. That's an additional developing step needed to allow
the developer to use tracepoints in his/her favorite language. But there
are other advantages, like generating tracepoints in two different
languages from the same tracepoint definitions. Like Mathieu wrote, it's
the same spirit as Flex and Bison. With Bison, you don't use the C
preprocessor to write your grammar rules: the tool reads a Bison input file
and generates a parser source file. Just like barectf reads a YAML
configuration file which defines stream and event classes and generates
tracer source files.

Phil

>
>
> because Go seems to be similar to C i'm strongly tempted to just co-opt
> the C pre-processor and existing LTTng tracepoint machinery and modify the
> LTTng header files to generate valid Go code for a tracepoint definition.
> this does add a LTTng tracepoint tool to the steps required to compile Go,
> except that the tracepoint tool is called "cpp" (plus header files).
>
> i don't really know Go yet and i hae not been assigned to get LTTng to
> work in Go, so this is all quite preliminary.  but i think the project i'm
> about to start on really needs LTTng or LTTng-like tracepoint functionality
> so i'm motivated to see if this can work.
>

> i'm going to investigate some more.
>
> --craig
>
>
> On Fri, Jun 2, 2017 at 10:05 AM, Philippe Proulx 
> wrote:
>
>> On Fri, Jun 2, 2017 at 11:58 AM, Mathieu Desnoyers <
>> mathieu.desnoy...@efficios.com> wrote:
>>
>>>
>>> - On Jun 2, 2017, at 2:27 AM, craig harmer 
>>> wrote:
>>>
>>> hi all,
>>>
>>> has anyone looked at porting LTTng to Go
>>>  (AKA
>>> Golang )?
>>>
>>>
>>> Not yet ! I'm glad someone is looking into it. :)
>>>
>>>
>>>
>>> Go is a language very similar to C, but it has intrinsic support for
>>> concurrency and better memory protection (at the expense of including
>>> garbage 

Re: [lttng-dev] porting LTTng to Go(lang)?

[craig harmer]

hi all,

the suggestion to write a pre-processor for LTTng tracepoints is
interesting and may be a good idea.  however, i'm not sure that it will
provide a general purpose solution.

the problem i for see is that tracepoint invocations, at least, should be
written in the syntax of the language they're being used in (a programmer
would would expect the tracepoint() function call to look like any other
function call in the language).

so the tracepoint parser needs to understand the function invocation syntax
of each language.  it also needs to understand at least some of the rest of
the syntax of the language -- in the case of Go it at least needs to be
able recognize "import" statements and import those files and it needs to
understand when something that looks like a tracepoint() is actually
embedded in a comment or string.  it also needs to emit code that matches
the syntax of the language.

so a general purpose lttng tracepoint parsing tool would need at least a
limited understanding of the syntax and grammar of each language for
reading the source file in and then writing the source file out.  there
could be code in the middle that's common across languages, but probably
not a lot.  perhaps an lttng tracepoint parsing tool could be better
structured as a common framework for language specific tracepoint parsers
to be plugged in to.

but writing a parser that actually understands the grammar of a language is
not easy.  if i was really going to do this, i'd be tempted to see if i
could just take this from gcc (which has front-ends that supports C, C++,
Go, and other languages (i think)).

and then there's the tracepoint definitions.  they could be written using
the barectf() tool that Philippe mentions, but i'm not sure if that would
be satisfactory for most programmers, who would prefer to write tracepoint
definitions in the same language that they are programming in.  they would
also need to support the data types of the language (i'm thinking Python
here as an example, where tracepoints would be expected to support most of
the Python basic data types).


because Go seems to be similar to C i'm strongly tempted to just co-opt the
C pre-processor and existing LTTng tracepoint machinery and modify the
LTTng header files to generate valid Go code for a tracepoint definition.
this does add a LTTng tracepoint tool to the steps required to compile Go,
except that the tracepoint tool is called "cpp" (plus header files).

i don't really know Go yet and i hae not been assigned to get LTTng to work
in Go, so this is all quite preliminary.  but i think the project i'm about
to start on really needs LTTng or LTTng-like tracepoint functionality so
i'm motivated to see if this can work.

i'm going to investigate some more.

--craig


On Fri, Jun 2, 2017 at 10:05 AM, Philippe Proulx 
wrote:

> On Fri, Jun 2, 2017 at 11:58 AM, Mathieu Desnoyers <
> mathieu.desnoy...@efficios.com> wrote:
>
>>
>> - On Jun 2, 2017, at 2:27 AM, craig harmer 
>> wrote:
>>
>> hi all,
>>
>> has anyone looked at porting LTTng to Go
>>  (AKA Golang
>> )?
>>
>>
>> Not yet ! I'm glad someone is looking into it. :)
>>
>>
>>
>> Go is a language very similar to C, but it has intrinsic support for
>> concurrency and better memory protection (at the expense of including
>> garbage collection).  its from Google and is pretty hot these days for big
>> data and "web scale" distributed systems.  the similarity to C is not
>> surprising since two of the three authors were also involved in the
>> development of UNIX in at AT in the 1970's: Rob Pike and Ken Thompson
>> (Robert Griesemer is the third author).
>>
>> Go is able to link with C object files, so implementing support for
>> tracef(3) should not be very difficult.
>>
>> but i want to have full support for user-defined tracepoint() events.
>> that looks to be much trickier since the LTTng tracepoint() functionality
>> relies heavily on (some might say abuses
>>
>> "abuses" is appropriate ;-)
>
>> ) the C pre-processor, and Go does not have an equivalent of the C
>> pre-processor.  however, i don't see an obvious reason why Go source files
>> that contain tracepoints couldn't be run through the C pre-processor --
>> except that Go specific LTTng header files would be required.
>>
>> here is an example of "hello, world" in Go
>> .  (note the use of "import" rather
>> then "#include").
>>
>> i've spent more than a few hours looking at the C code generated by the C
>> pre-processor trying to track down exactly why a tracepoint() definition is
>> causing a compiler error, so i think i have an idea of just how hard it
>> would be to development equivalent header files for Go + C pre-processor.
>> it may be the case that the Go "header files", once developed, would be
>> compiler specific, i.e only work with gccgo
>> 

Re: [lttng-dev] porting LTTng to Go(lang)?

[Mathieu Desnoyers]

- On Jun 2, 2017, at 2:27 AM, craig harmer  wrote: 

> hi all,

> has anyone looked at porting LTTng to [
> https://en.wikipedia.org/wiki/Go_%28programming_language%29 | Go ] (AKA [
> https://golang.org/ | Golang ] )?

Not yet ! I'm glad someone is looking into it. :) 

> Go is a language very similar to C, but it has intrinsic support for 
> concurrency
> and better memory protection (at the expense of including garbage collection).
> its from Google and is pretty hot these days for big data and "web scale"
> distributed systems. the similarity to C is not surprising since two of the
> three authors were also involved in the development of UNIX in at AT in the
> 1970's: Rob Pike and Ken Thompson (Robert Griesemer is the third author).

> Go is able to link with C object files, so implementing support for tracef(3)
> should not be very difficult.

> but i want to have full support for user-defined tracepoint() events. that 
> looks
> to be much trickier since the LTTng tracepoint() functionality relies heavily
> on (some might say abuses) the C pre-processor, and Go does not have an
> equivalent of the C pre-processor. however, i don't see an obvious reason why
> Go source files that contain tracepoints couldn't be run through the C
> pre-processor -- except that Go specific LTTng header files would be required.

> [ https://tour.golang.org/welcome/1 | here is an example of "hello, world" in 
> Go
> ] . (note the use of "import" rather then "#include").

> i've spent more than a few hours looking at the C code generated by the C
> pre-processor trying to track down exactly why a tracepoint() definition is
> causing a compiler error, so i think i have an idea of just how hard it would
> be to development equivalent header files for Go + C pre-processor. it may be
> the case that the Go "header files", once developed, would be compiler
> specific, i.e only work with [ https://golang.org/doc/install/gccgo | gccgo ]
> (the gcc front-end for Go).

> anyway, i'm wondering if anybody has attempted this and/or what your thoughts
> would be.

In the case of C/C++ code, using the C preprocessor got the job done without 
requiring any additional 
dependency. Given the context you describe, perhaps it would be wise to 
consider introducing a dedicated 
"lttng probe description" parser. The idea here would be to parse the 
tracepoint probe definitions and 
translate those into native code for various languages (e.g. Golang). 

This would create a new lttng utility that would be required to to produce the 
language-specific files from 
those descriptions. I would allow us to port the "tracepoint" concept to many 
more languages easily, and 
not be so much tied to the C preprocessor anymore. 

So perhaps adding this new tool as a dependency that needs to be invoked prior 
to compilation might 
not be too much of an issue ? I see it as being slightly similar to the role 
accomplished by Flex and 
Bison: they are required to translate from source files to an intermediary 
language, and then the 
resulting files can be included into the distribution source packages, so only 
those who aim at 
changing the source descriptions need to have the translation tool installed. 

Thoughts ? 

Thanks, 

Mathieu 

> --craig

> ___
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev