Re: [PATCH 0/2] Python testcases to check DWARF output

[Iain Buclaw]

On 4 August 2017 at 00:23, Mike Stump  wrote:
> On Jul 27, 2017, at 2:07 AM, Pierre-Marie de Rodat  
> wrote:
>> On 07/27/2017 09:52 AM, Richard Biener wrote:
 I'm fine with the direction if a reviewer wants to go in that
 direction.  I wish python didn't have a built-in speed penalty,
 that's the only downside I don't like about it.  Aside from that,
 even switching all of the testsuite to be python based isn't a
 terrible idea.
>>> But is it worse than TCL?
>
> python is likely 2x faster than tcl, if you have one instance per testsuite 
> run.  The problem is, for the work required, it's cheaper to do the work once 
> to cut over to a new language.  I'd rather switch to some other language that 
> can come closer to the speed of compiled C code, yet in the scripting family. 
>  I don't have a pointer to a better solution than python at this time.  I'd 
> not be opposed to switching to python, it should be faster, just as safe, a 
> bit easier for new people to code in, and more people who know it and use it. 
>  I think python is funner to code in than tcl.  Cutting the entire testsuite 
> over at once, might well be more than any one person can contribute, but, we 
> could cut over subtrees, as people donate time; if people want to go in that 
> direction.  This can't be a 1 person decision, but rather a consensus 
> building type decision.  What do others think?
>

Sounds good to me.  Having recently done quite a bit of writing in tcl
for the D testsuite recently, there are certainly many gotchas that I
ran into.  Not sure whether you would rather take an existing testing
framework, or write one internally, but I can imagine something that
makes heavy use of python decorators for the simple cases.

Re: [PATCH 0/2] Python testcases to check DWARF output

[Mike Stump]

On Jul 27, 2017, at 2:07 AM, Pierre-Marie de Rodat  wrote:
> On 07/27/2017 09:52 AM, Richard Biener wrote:
>>> I'm fine with the direction if a reviewer wants to go in that
>>> direction.  I wish python didn't have a built-in speed penalty,
>>> that's the only downside I don't like about it.  Aside from that,
>>> even switching all of the testsuite to be python based isn't a
>>> terrible idea.
>> But is it worse than TCL?

python is likely 2x faster than tcl, if you have one instance per testsuite 
run.  The problem is, for the work required, it's cheaper to do the work once 
to cut over to a new language.  I'd rather switch to some other language that 
can come closer to the speed of compiled C code, yet in the scripting family.  
I don't have a pointer to a better solution than python at this time.  I'd not 
be opposed to switching to python, it should be faster, just as safe, a bit 
easier for new people to code in, and more people who know it and use it.  I 
think python is funner to code in than tcl.  Cutting the entire testsuite over 
at once, might well be more than any one person can contribute, but, we could 
cut over subtrees, as people donate time; if people want to go in that 
direction.  This can't be a 1 person decision, but rather a consensus building 
type decision.  What do others think?

> Good point. Actually for Python there are ways to make it faster. If we can 
> somehow manage to have a limited set of Python interpreter instances (instead 
> of one per test), we could use pypy, which is very good I heard to make long 
> running instances fast.

Yes.  One instance would help ensure the performance is good.  I don't have a 
firm grasp of startup time to know just how critical it is.  Also, I don't have 
a good grasp on memory pressures that python would create, say, compared to how 
we use tcl.

Re: [PATCH 0/2] Python testcases to check DWARF output

[Jeff Law]

On 08/03/2017 02:27 AM, Pierre-Marie de Rodat wrote:
> On 08/02/2017 05:43 PM, Jeff Law wrote:
>> I hate to throw in a wrench at this point, but has anyone looked at
>> dwgrep from Petr Machata?  He's not doing much with it anymore, but it
>> might provide enough of a dwarf scanning framework to be useful for
>> testing purposes.
> 
> Sure, no problem: I first started talking publicly about this one week
> ago, so it’s definitely not too late to mention alternatives. ;-) I
> learned about dwgrep two years ago and forgot about it, so thank you for
> the idea. I started to have a look at it, and for now I don’t think it’s
> a good match in this context:
> 
>  1. it’s an ELF only tool;
>  2. it must be built, requiring external dependencies: cmake and
> elfutils;
>  3. in order to use it, one must learn a dedicated post-fix language
> (Zwerg)
> 
> For 1. I think this is a true problem, as it means for instance that we
> could not test DWARF on Windows and Darwin setups. Unless we add PE and
> Mach-O handling in dwgrep of course, but that does not sound easy and
> will bring other external dependencies.
> 
> For 3. I feel that, for someone who is comfortable with Python, it will
> be easier to deal with a Python library (the dwarfutils in my patch)
> than having to learn yet another DSL. I think that’s precisely why some
> people would like to have a Python test framework rather than a TCL one.
> Working with a “usual” imperative language looks easier than with
> postfix expressions. Smaller cognitive load.
> 
> Actually I see another problem: pattern will have to vary depending on
> the target platform (for instance 32/64bit or depending on the DWARF
> version). Of course we could duplicate whole patterns in testcases to
> take this into account, but that’s like code duplication: I think we
> should be able to include small “X if 32bit else Y” in patterns, and I
> don’t think we can do that with Zwerg (no way to pass something like
> environment variables).
> 
> Of course, I have written a “competitor” tool: I guess my judgment is
> biased. :-) So other opinions are welcome!
Thanks.  I just wanted to raise it as a possibility.  It looks like it's
not a good fit here, so let's not let it be a distraction.

jeff

Re: [PATCH 0/2] Python testcases to check DWARF output

[Pierre-Marie de Rodat]

On 08/02/2017 05:43 PM, Jeff Law wrote:

I hate to throw in a wrench at this point, but has anyone looked at
dwgrep from Petr Machata?  He's not doing much with it anymore, but it
might provide enough of a dwarf scanning framework to be useful for
testing purposes.


Sure, no problem: I first started talking publicly about this one week 
ago, so it’s definitely not too late to mention alternatives. ;-) I 
learned about dwgrep two years ago and forgot about it, so thank you for 
the idea. I started to have a look at it, and for now I don’t think it’s 
a good match in this context:


 1. it’s an ELF only tool;
 2. it must be built, requiring external dependencies: cmake and
elfutils;
 3. in order to use it, one must learn a dedicated post-fix language
(Zwerg)

For 1. I think this is a true problem, as it means for instance that we 
could not test DWARF on Windows and Darwin setups. Unless we add PE and 
Mach-O handling in dwgrep of course, but that does not sound easy and 
will bring other external dependencies.


For 3. I feel that, for someone who is comfortable with Python, it will 
be easier to deal with a Python library (the dwarfutils in my patch) 
than having to learn yet another DSL. I think that’s precisely why some 
people would like to have a Python test framework rather than a TCL one. 
Working with a “usual” imperative language looks easier than with 
postfix expressions. Smaller cognitive load.


Actually I see another problem: pattern will have to vary depending on 
the target platform (for instance 32/64bit or depending on the DWARF 
version). Of course we could duplicate whole patterns in testcases to 
take this into account, but that’s like code duplication: I think we 
should be able to include small “X if 32bit else Y” in patterns, and I 
don’t think we can do that with Zwerg (no way to pass something like 
environment variables).


Of course, I have written a “competitor” tool: I guess my judgment is 
biased. :-) So other opinions are welcome!

--
Pierre-Marie de Rodat

Re: [PATCH 0/2] Python testcases to check DWARF output

[Jeff Law]

On 07/27/2017 01:52 AM, Richard Biener wrote:
> On Wed, Jul 26, 2017 at 11:25 PM, Mike Stump  wrote:
>> On Jul 26, 2017, at 9:00 AM, Pierre-Marie de Rodat  
>> wrote:
>>> At the last GNU Cauldron, Richard Biener and I talked about DWARF output
>>> testing. Except for guality tests, which are disabled on several
>>> targets, the only way tests check the DWARF is scanning the annotated
>>> assembly (-dA), making it hard to write reliable tests.
>>
>>> Anyway, Richard and I discussed about doing something similar in-tree,
>>> and here is a candidate set of patches to achieve that
>>
>> I'm fine with the direction if a reviewer wants to go in that direction.  I 
>> wish python didn't have a built-in speed penalty, that's the only downside I 
>> don't like about it.  Aside from that, even switching all of the testsuite 
>> to be python based isn't a terrible idea.
> 
> But is it worse than TCL?
I don't think Python is worse than TCL.  Few things would have that
property.  And I think we're a lot more likely to be able to find folks
that can hack Python as needed vs hacking TCL.

jeff

Re: [PATCH 0/2] Python testcases to check DWARF output

[Jeff Law]

On 07/26/2017 10:00 AM, Pierre-Marie de Rodat wrote:
> Hello,
> 
> At the last GNU Cauldron, Richard Biener and I talked about DWARF output
> testing. Except for guality tests, which are disabled on several
> targets, the only way tests check the DWARF is scanning the annotated
> assembly (-dA), making it hard to write reliable tests.
> 
> For instance, checking the number of times DW_AT_location is present in
> order to check that some specific variable is assigned one is fuzzy.
> Depending on the target and on the evolution of the compiler, the number
> of output variables, or which one is assigned a location can vary
> legitimately but still make the test fail.
> 
> On my side, I already had written an out-of-tree testsuite for the DWARF
> features I added for Ada. This testsuite uses a DWARF parser in order to
> perform checks on a tree:
> . I had to update it
> a couple of times, for instance when a change created a
> DW_TAG_const_type DIE or removed one somewhere in a type tree, but
> that’s very rare. I would say that I’m satisfied with the checks I could
> express, but I don’t remember I ever caught a regression with them, so I
> have no representative experience to share in this area. Maybe DWARF
> back-end developpers do a too good job. ;-)
> 
> Anyway, Richard and I discussed about doing something similar in-tree,
> and here is a candidate set of patches to achieve that:
> 
>   * The first patch installs DejaGNU scripts to run a Python interpreter
> in testcases.
> 
>   * The second one installs other DejaGNU scripts to detect DWARF
> dumping tools, plus a small Python library to parse and pattern
> match DIEs and their attributes. It also adds several C and Ada
> tests as examples; these are inspired by existing homonym tests
> based on assembly scanning.
> 
> For now, this supports only platforms where objdump is available for the
> current target, but extending it to other tools, such as otool on Darwin
> should be doable.
> 
> I would appreciate feedback about the idea and the implementation I
> propose. This is the first time I do more in the testsuite than just
> adding new tests, so thank you in advance for you patience in reviewing
> these. :-)
> 
> I tested these patches on x86_64-linux.
I hate to throw in a wrench at this point, but has anyone looked at
dwgrep from Petr Machata?  He's not doing much with it anymore, but it
might provide enough of a dwarf scanning framework to be useful for
testing purposes.

jeff

Re: [PATCH 0/2] Python testcases to check DWARF output

[Pierre-Marie de Rodat]

Thank you for your feedback.

On 07/27/2017 09:52 AM, Richard Biener wrote:

I'm fine with the direction if a reviewer wants to go in that
direction.  I wish python didn't have a built-in speed penalty,
that's the only downside I don't like about it.  Aside from that,
even switching all of the testsuite to be python based isn't a
terrible idea.


But is it worse than TCL?


Good point. Actually for Python there are ways to make it faster. If we 
can somehow manage to have a limited set of Python interpreter instances 
(instead of one per test), we could use pypy, which is very good I heard 
to make long running instances fast.


As to switch all of the testsuite to Python, I don’t have an educated 
opinion on this. I just want to say that, here I’m using Python to 
pattern match DIEs, but if needed we could perfectly use it to do other 
complex tasks. This is why I kept the DWARF-specific stuff 
(gcc-dwarf.exp and the dwarfutils Python package, from second commit) 
separate from just Python interpreter handling (gcc-python.exp, from 
first commit).


Note that having a Python only testsuite would make it easier to have 
only one Python instance for all the testsuite run, so it would, in 
theory, make it easier to get a fast execution.


… now, I’m not familiar with DejaGNU but I have the feeling that it does 
a lot with respect to the handling of a great variety of 
targets/remote/etc. combinations. Re-writing it (and making sure it 
works!) sounds like a huuuge task. I’ll let experts in this area 
comment. :-)


--
Pierre-Marie de Rodat

Re: [PATCH 0/2] Python testcases to check DWARF output

[Richard Biener]

On Wed, Jul 26, 2017 at 11:25 PM, Mike Stump  wrote:
> On Jul 26, 2017, at 9:00 AM, Pierre-Marie de Rodat  
> wrote:
>> At the last GNU Cauldron, Richard Biener and I talked about DWARF output
>> testing. Except for guality tests, which are disabled on several
>> targets, the only way tests check the DWARF is scanning the annotated
>> assembly (-dA), making it hard to write reliable tests.
>
>> Anyway, Richard and I discussed about doing something similar in-tree,
>> and here is a candidate set of patches to achieve that
>
> I'm fine with the direction if a reviewer wants to go in that direction.  I 
> wish python didn't have a built-in speed penalty, that's the only downside I 
> don't like about it.  Aside from that, even switching all of the testsuite to 
> be python based isn't a terrible idea.

But is it worse than TCL?

Richard.

Re: [PATCH 0/2] Python testcases to check DWARF output

[Mike Stump]

On Jul 26, 2017, at 9:00 AM, Pierre-Marie de Rodat  wrote:
> At the last GNU Cauldron, Richard Biener and I talked about DWARF output
> testing. Except for guality tests, which are disabled on several
> targets, the only way tests check the DWARF is scanning the annotated
> assembly (-dA), making it hard to write reliable tests.

> Anyway, Richard and I discussed about doing something similar in-tree,
> and here is a candidate set of patches to achieve that

I'm fine with the direction if a reviewer wants to go in that direction.  I 
wish python didn't have a built-in speed penalty, that's the only downside I 
don't like about it.  Aside from that, even switching all of the testsuite to 
be python based isn't a terrible idea.

smime.p7s
Description: S/MIME cryptographic signature

Re: [PATCH 0/2] Python testcases to check DWARF output

[Pierre-Marie de Rodat]

On 07/26/2017 06:15 PM, David Malcolm wrote:

(FWIW I'm a big fan of Python, so am happy to see this proposal)


Me too. :-)


Which version of Python did you test against?   As far as I can see
you've coded this using the common subset of Python 2 and Python 3;
it's worth spelling out what the assumptions are in this regard (and
what the minimum versions are).


I tested with both Python 2 and Python 3, as “python” can be each one 
depending on the system (it’s Python 3 on my Linux box). I agree I 
should explicitely state that sources must be compatible with both. 
Maybe in lib/gcc-python.exp?


--
Pierre-Marie de Rodat

Re: [PATCH 0/2] Python testcases to check DWARF output

[David Malcolm]

On Wed, 2017-07-26 at 18:00 +0200, Pierre-Marie de Rodat wrote:
> Hello,
> 
> At the last GNU Cauldron, Richard Biener and I talked about DWARF
> output
> testing. Except for guality tests, which are disabled on several
> targets, the only way tests check the DWARF is scanning the annotated
> assembly (-dA), making it hard to write reliable tests.
> 
> For instance, checking the number of times DW_AT_location is present
> in
> order to check that some specific variable is assigned one is fuzzy.
> Depending on the target and on the evolution of the compiler, the
> number
> of output variables, or which one is assigned a location can vary
> legitimately but still make the test fail.
> 
> On my side, I already had written an out-of-tree testsuite for the
> DWARF
> features I added for Ada. This testsuite uses a DWARF parser in order
> to
> perform checks on a tree:
> . I had to update
> it
> a couple of times, for instance when a change created a
> DW_TAG_const_type DIE or removed one somewhere in a type tree, but
> that’s very rare. I would say that I’m satisfied with the checks I
> could
> express, but I don’t remember I ever caught a regression with them,
> so I
> have no representative experience to share in this area. Maybe DWARF
> back-end developpers do a too good job. ;-)
> 
> Anyway, Richard and I discussed about doing something similar in
> -tree,
> and here is a candidate set of patches to achieve that:
> 
>   * The first patch installs DejaGNU scripts to run a Python
> interpreter
> in testcases.
> 
>   * The second one installs other DejaGNU scripts to detect DWARF
> dumping tools, plus a small Python library to parse and pattern
> match DIEs and their attributes. It also adds several C and Ada
> tests as examples; these are inspired by existing homonym tests
> based on assembly scanning.
> 
> For now, this supports only platforms where objdump is available for
> the
> current target, but extending it to other tools, such as otool on
> Darwin
> should be doable.
> 
> I would appreciate feedback about the idea and the implementation I
> propose. This is the first time I do more in the testsuite than just
> adding new tests, so thank you in advance for you patience in
> reviewing
> these. :-)

(FWIW I'm a big fan of Python, so am happy to see this proposal)

> I tested these patches on x86_64-linux.

Which version of Python did you test against?   As far as I can see
you've coded this using the common subset of Python 2 and Python 3;
it's worth spelling out what the assumptions are in this regard (and
what the minimum versions are).

Dave