Re: Coverage analysis update

[Joel Sherrill]

On Fri, Jul 7, 2017 at 10:27 AM, Cillian O'Donnell 
wrote:

> On 7 July 2017 at 03:58, Joel Sherrill  wrote:
> >
> >
> > On Jul 6, 2017 9:20 PM, "Chris Johns"  wrote:
> >
> > On 07/07/2017 12:10, Joel Sherrill wrote:
> >>
> >>
> >> On Jul 6, 2017 8:52 PM, "Chris Johns"  >> > wrote:
> >>
> >> On 07/07/2017 00:34, Joel Sherrill wrote:
> >> > On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell
> >>  >> 
> >> > >>
> >> wrote:
> >> > It will ignore records when it thinks things are inconsistent.
> This
> >> can occur
> >> > when a method appears in two different executables and has
> different
> >> > sizes. The cause of this is usually padding at the end of the
> method
> >> so
> >> > the subsequent method in memory starts on a nice cache-line
> >> alignment.
> >> > The code tries to recognize the nops/padding at the end and ignore
> >> them.
> >>
> >> The code in the linked executable can be different to the object
> file.
> >> The
> >> linker does different things on different architectures and
> different
> >> link
> >> orders.
> >>
> >>
> >> We do not look at the .o files. The objdump output on exe files is used
> >
> > I would like to see this changed to use:
> >
> > https://git.rtems.org/rtems-tools/tree/rtemstoolkit/rld-symbols.h#n224
> >
> > and to remove any objdump access. The toolkit will be much faster at
> loading
> > a
> > symbol table.
> >
> >
> > That could replace the use of NM. Could it also replace objdump?
> >
> > It looks like now that he has problems with the Couverture trace and
> running
> > in parallel which is independent.
>
> I ran rtems-test single threaded --jobs=1 and it locked up on the
> first executable base_sp.exe of the covoar run.
>
>   information->inactive_per_block[ block ] = information->allocation_size;
>  4000c154:   c4 16 20 14 lduh  [ %i0 + 0x14 ], %g2
>  4000c158:   c6 06 20 30 ld  [ %i0 + 0x30 ], %g3
>
> Just stops here with no error message. So maybe the IO Error is about
> parallel tasks, I'd have to run it a few more times to be sure.
>

This looks like qemu isn't completing the execution of the RTEMS
application.

>
> Also I did compare qemu-traces.h with couverture's and there was a few
> changes to be made. It matches exactly now and I rebuilt covoar. It
> still lock's up as above though.
>
> Also I tried to build qemu-dump-traces like you mentioned Joel waf
> builds and doesn't complain about anything but there's no executable
> in the build directory, should there be? How does qemy-dump-traces
> work? The waf build looks like this:
>
> 127 bld.program(target = 'qemu-dump-trace',
> 128 source = ['qemu-dump-trace.c'],
> 129 use = 'ccovoar',
> 130 cflags = ['-O2', '-g'],
> 131 includes = ['.'])
>
>
When you get a qemu-dump-trace executable, it is invoked like

/qemu-dump-trace tracefile1... tracefileN

This is good to see if the information looks reasonable given the program
you
are looking at the trace of. It is a very simple program and does **NOT**
contain any source code from covoar so it doesn't prove that covoar is
reading the trace correctly -- just that the basic look of the trace file
is OK
given the hand inspection.


> >
> > As a test, can you run all tests with coverage disabled?
>
> Yes the full testsuite runs fine with coverage disabled, it's only the
> post testsuite covoar runs that ever run into trouble
>

If you comment out the execution of covoar in rtems-tester does it still
hang.
Everything you post shows a SPARC trace which isn't from covoar execution.
Just print "would run covoar here and show the command line" :)

We need to be sure that an rtems-tester run completes with couverture
with both coverage tracing on and off. Without it invoking covoar.

Then you can run covoar by hand after using the output.


> >
> > It would be helpful if running the tests and running covoar could be
> > separate. Easier to debug.
>
> This would be handy to not have to wait for the full testsuite to run
> every time to check if something has worked or not. I'm not sure how
> to separate the covoar part though.
>

Save the configuration file from a single covoar run and print the
command line. You can save a build tree, trace files, etc. and just
run covoar by hand repeatedly.


> >
> >
> >> > When the padding inserted by ld changes or the objdump output
> being>
> >> parsed changes, covoar needs to be adjusted.
> >>
> >> This means fragile.
> >>
> >>
> >> Yes a bit.  It has to be taught by architecture what padding LD puts in.
> >> But
> >> this doesn't change often or for no reason.
> >>
> >> Ian Taylor assured me years ago that the objdump output format was 5he
> >> most
> >> stable way to do this.
> >
> > 

Re: Coverage analysis update

[Cillian O'Donnell]

On 7 July 2017 at 02:39, Chris Johns  wrote:
> On 07/07/2017 00:34, Joel Sherrill wrote:
>> On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell > > wrote:
>> >> When a test run is successful it will also finish with a lot of these 
>> messages:
>> Also I wanted to mention, there is quite a bit of work from the 2
>> previous students on the integration of the coverage work into RTEMS
>> Tester, would you be interested in starting the merging process of
>> this. It might take a while to get everything reviewed and make any
>> necessary changes. At this point it is working. its just fixing bugs,
>> which I'm working on at the moment.
>>
>> Chris.. I suggested this. It seems the basic flow and integration of
>> coverage into the tester is good enough to review and merge.
>>
>
> Is there a place where this work is merged into something that can be 
> reviewed?

I just rebased against master and the work can be found under the
coverage-review branch here if you want to take a look:

https://github.com/cillianodonnell/rtems-tools/tree/coverage-review

Thanks,

Cillian.
>
> Is there an explanation of the procedure being implemented? The last I could
> find is https://lists.rtems.org/pipermail/devel/2014-May/006692.html, search 
> for
> "The procedure is as follows ...".
>
> For merging I am interested in the merging of what may be happening externally
> in scripts to use the functionality provided in the RTEMS Toolkit, for example
> symbols.
>
> Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Coverage analysis update

[Cillian O'Donnell]

On 7 July 2017 at 03:58, Joel Sherrill  wrote:
>
>
> On Jul 6, 2017 9:20 PM, "Chris Johns"  wrote:
>
> On 07/07/2017 12:10, Joel Sherrill wrote:
>>
>>
>> On Jul 6, 2017 8:52 PM, "Chris Johns" > > wrote:
>>
>> On 07/07/2017 00:34, Joel Sherrill wrote:
>> > On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell
>> > 
>> > >>
>> wrote:
>> > It will ignore records when it thinks things are inconsistent. This
>> can occur
>> > when a method appears in two different executables and has different
>> > sizes. The cause of this is usually padding at the end of the method
>> so
>> > the subsequent method in memory starts on a nice cache-line
>> alignment.
>> > The code tries to recognize the nops/padding at the end and ignore
>> them.
>>
>> The code in the linked executable can be different to the object file.
>> The
>> linker does different things on different architectures and different
>> link
>> orders.
>>
>>
>> We do not look at the .o files. The objdump output on exe files is used
>
> I would like to see this changed to use:
>
> https://git.rtems.org/rtems-tools/tree/rtemstoolkit/rld-symbols.h#n224
>
> and to remove any objdump access. The toolkit will be much faster at loading
> a
> symbol table.
>
>
> That could replace the use of NM. Could it also replace objdump?
>
> It looks like now that he has problems with the Couverture trace and running
> in parallel which is independent.

I ran rtems-test single threaded --jobs=1 and it locked up on the
first executable base_sp.exe of the covoar run.

  information->inactive_per_block[ block ] = information->allocation_size;
 4000c154:   c4 16 20 14 lduh  [ %i0 + 0x14 ], %g2
 4000c158:   c6 06 20 30 ld  [ %i0 + 0x30 ], %g3

Just stops here with no error message. So maybe the IO Error is about
parallel tasks, I'd have to run it a few more times to be sure.

Also I did compare qemu-traces.h with couverture's and there was a few
changes to be made. It matches exactly now and I rebuilt covoar. It
still lock's up as above though.

Also I tried to build qemu-dump-traces like you mentioned Joel waf
builds and doesn't complain about anything but there's no executable
in the build directory, should there be? How does qemy-dump-traces
work? The waf build looks like this:

127 bld.program(target = 'qemu-dump-trace',
128 source = ['qemu-dump-trace.c'],
129 use = 'ccovoar',
130 cflags = ['-O2', '-g'],
131 includes = ['.'])

>
> As a test, can you run all tests with coverage disabled?

Yes the full testsuite runs fine with coverage disabled, it's only the
post testsuite covoar runs that ever run into trouble
>
> It would be helpful if running the tests and running covoar could be
> separate. Easier to debug.

This would be handy to not have to wait for the full testsuite to run
every time to check if something has worked or not. I'm not sure how
to separate the covoar part though.
>
>
>> > When the padding inserted by ld changes or the objdump output being>
>> parsed changes, covoar needs to be adjusted.
>>
>> This means fragile.
>>
>>
>> Yes a bit.  It has to be taught by architecture what padding LD puts in.
>> But
>> this doesn't change often or for no reason.
>>
>> Ian Taylor assured me years ago that the objdump output format was 5he
>> most
>> stable way to do this.
>
> We have direct access to the EFL file and symbols. I see that as a better
> solution.
>
>
> When things work. I think there are two issues ahead of any issues because
> of this. It has been on my wishlist but we need (1) to be to run all tests
> successfully and (2) to be sure the Couverture trace is read correctly.
>
>
>> > The ignored record message I saw is in the code that reads
>> Couverture
>> > trace records. The info in the record appears to be inconsistent
>> with the
>> > code to which it has been matched.
>>
>> Sorry, I do not understand why this difference is happening? I
>> understand it is
>> object files vs executable, what I do not understand is why the object
>> files are
>> being referenced, why not just use the executable?
>>
>>
>> No .o is used. We haven't parsed couverture trace format in years. It
>> could have
>> changed. I **think** the message indicates that the qemu translation block
>> is
>> reported as longer than from the current instruction to the end of the
>> method.
>>
>> The answer is to know the address range of the flagged trace block and
>> what it
>> corresponds to in the exe.
>
> I wonder if ELF holds the size of the area or can the symbol table sorted by
> address produce the needed ranges?
>
>
> I do not know. I think it has the range but it would be straightforward I
> think to replace the use of NM.
>
>
>> Fwiw this thread 

Re: Coverage analysis update

[Chris Johns]

On 07/07/2017 12:58, Joel Sherrill wrote:
> On Jul 6, 2017 9:20 PM, "Chris Johns"  > wrote:
> 
> On 07/07/2017 12:10, Joel Sherrill wrote:
> >
> >
> > On Jul 6, 2017 8:52 PM, "Chris Johns"  
> > >> wrote:
> >
> > On 07/07/2017 00:34, Joel Sherrill wrote:
> > > On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell
> 
> > >
> > > 
>  > > It will ignore records when it thinks things are inconsistent. 
> This
> can occur
> > > when a method appears in two different executables and has 
> different
> > > sizes. The cause of this is usually padding at the end of the 
> method so
> > > the subsequent method in memory starts on a nice cache-line 
> alignment.
> > > The code tries to recognize the nops/padding at the end and 
> ignore them.
> >
> > The code in the linked executable can be different to the object 
> file. The
> > linker does different things on different architectures and 
> different link
> > orders.
> >
> >
> > We do not look at the .o files. The objdump output on exe files is used
> 
> I would like to see this changed to use:
> 
> https://git.rtems.org/rtems-tools/tree/rtemstoolkit/rld-symbols.h#n224
> 
> 
> and to remove any objdump access. The toolkit will be much faster at 
> loading a
> symbol table.
> 
> 
> That could replace the use of NM. Could it also replace objdump?
> 

Oh nm is being used. What is objdump being used for?

> It looks like now that he has problems with the Couverture trace and running 
> in
> parallel which is independent.
> 
> As a test, can you run all tests with coverage disabled? 
> 
> It would be helpful if running the tests and running covoar could be separate.
> Easier to debug.
> 
> 
> > > When the padding inserted by ld changes or the objdump output 
> being>
> > parsed changes, covoar needs to be adjusted.
> >
> > This means fragile.
> >
> >
> > Yes a bit.  It has to be taught by architecture what padding LD puts 
> in. But
> > this doesn't change often or for no reason.
> >
> > Ian Taylor assured me years ago that the objdump output format was 5he 
> most
> > stable way to do this.
> 
> We have direct access to the EFL file and symbols. I see that as a better
> solution.
> 
> 
> When things work. I think there are two issues ahead of any issues because of
> this. 

I cannot tell, I do not know how this all works. I was wondering if the fewer
tools and files created the less room there is for error.

> It has been on my wishlist but we need (1) to be to run all tests
> successfully and (2) to be sure the Couverture trace is read correctly.
> 
> 
> > > The ignored record message I saw is in the code that reads 
> Couverture
> > > trace records. The info in the record appears to be inconsistent
> with the
> > > code to which it has been matched.
> >
> > Sorry, I do not understand why this difference is happening? I
> understand it is
> > object files vs executable, what I do not understand is why the 
> object
> files are
> > being referenced, why not just use the executable?
> >
> >
> > No .o is used. We haven't parsed couverture trace format in years. It
> could have
> > changed. I **think** the message indicates that the qemu translation 
> block is
> > reported as longer than from the current instruction to the end of the 
> method.
> >
> > The answer is to know the address range of the flagged trace block and 
> what it
> > corresponds to in the exe.
> 
> I wonder if ELF holds the size of the area or can the symbol table sorted 
> by
> address produce the needed ranges?
> 
> 
> I do not know. I think it has the range but it would be straightforward I 
> think
> to replace the use of NM.

It may in the DWARF info which we do not have access to.

> > Fwiw this thread needs to be split. There are multiple issues.
> 
> This specific fragment I have created to address symbols?
> 
> 
> If the issue is NM versus elf info, then it would help. Is there an RTEMS NM
> based on the library?

To load a symbol table you do this:

https://git.rtems.org/rtems-tools/tree/linkers/rtems-syms.cpp#n466

After that you have a symbol table you can use to get at the symbols.

Chris
___
devel mailing list
devel@rtems.org

Re: Coverage analysis update

[Chris Johns]

On 07/07/2017 12:10, Joel Sherrill wrote:
> 
> 
> On Jul 6, 2017 8:52 PM, "Chris Johns"  > wrote:
> 
> On 07/07/2017 00:34, Joel Sherrill wrote:
> > On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell  
> > >> wrote:
> > It will ignore records when it thinks things are inconsistent. This can 
> occur
> > when a method appears in two different executables and has different
> > sizes. The cause of this is usually padding at the end of the method so
> > the subsequent method in memory starts on a nice cache-line alignment.
> > The code tries to recognize the nops/padding at the end and ignore them.
> 
> The code in the linked executable can be different to the object file. The
> linker does different things on different architectures and different link
> orders.
> 
> 
> We do not look at the .o files. The objdump output on exe files is used

I would like to see this changed to use:

https://git.rtems.org/rtems-tools/tree/rtemstoolkit/rld-symbols.h#n224

and to remove any objdump access. The toolkit will be much faster at loading a
symbol table.

> > When the padding inserted by ld changes or the objdump output being>
> parsed changes, covoar needs to be adjusted.
> 
> This means fragile.
> 
> 
> Yes a bit.  It has to be taught by architecture what padding LD puts in. But
> this doesn't change often or for no reason.
> 
> Ian Taylor assured me years ago that the objdump output format was 5he most
> stable way to do this.

We have direct access to the EFL file and symbols. I see that as a better 
solution.

> > The ignored record message I saw is in the code that reads Couverture
> > trace records. The info in the record appears to be inconsistent with 
> the
> > code to which it has been matched.
> 
> Sorry, I do not understand why this difference is happening? I understand 
> it is
> object files vs executable, what I do not understand is why the object 
> files are
> being referenced, why not just use the executable?
> 
> 
> No .o is used. We haven't parsed couverture trace format in years. It could 
> have
> changed. I **think** the message indicates that the qemu translation block is
> reported as longer than from the current instruction to the end of the 
> method. 
> 
> The answer is to know the address range of the flagged trace block and what it
> corresponds to in the exe.

I wonder if ELF holds the size of the area or can the symbol table sorted by
address produce the needed ranges?

> Fwiw this thread needs to be split. There are multiple issues. 

This specific fragment I have created to address symbols?

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Coverage analysis update

[Joel Sherrill]

On Jul 6, 2017 8:52 PM, "Chris Johns"  wrote:

On 07/07/2017 00:34, Joel Sherrill wrote:
> On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell  > wrote:
> It will ignore records when it thinks things are inconsistent. This can
occur
> when a method appears in two different executables and has different
> sizes. The cause of this is usually padding at the end of the method so
> the subsequent method in memory starts on a nice cache-line alignment.
> The code tries to recognize the nops/padding at the end and ignore them.

The code in the linked executable can be different to the object file. The
linker does different things on different architectures and different link
orders.


We do not look at the .o files. The objdump output on exe files is used


> When the padding inserted by ld changes or the objdump output being>
parsed changes, covoar needs to be adjusted.

This means fragile.


Yes a bit.  It has to be taught by architecture what padding LD puts in.
But this doesn't change often or for no reason.

Ian Taylor assured me years ago that the objdump output format was 5he most
stable way to do this.


> The ignored record message I saw is in the code that reads Couverture
> trace records. The info in the record appears to be inconsistent with the
> code to which it has been matched.

Sorry, I do not understand why this difference is happening? I understand
it is
object files vs executable, what I do not understand is why the object
files are
being referenced, why not just use the executable?


No .o is used. We haven't parsed couverture trace format in years. It could
have changed. I **think** the message indicates that the qemu translation
block is reported as longer than from the current instruction to the end of
the method.

The answer is to know the address range of the flagged trace block and what
it corresponds to in the exe.

Fwiw this thread needs to be split. There are multiple issues.


Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Coverage analysis update

[Chris Johns]

On 07/07/2017 00:34, Joel Sherrill wrote:
> On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell  > wrote:
> It will ignore records when it thinks things are inconsistent. This can occur
> when a method appears in two different executables and has different
> sizes. The cause of this is usually padding at the end of the method so
> the subsequent method in memory starts on a nice cache-line alignment.
> The code tries to recognize the nops/padding at the end and ignore them.

The code in the linked executable can be different to the object file. The
linker does different things on different architectures and different link 
orders.

> When the padding inserted by ld changes or the objdump output being> parsed 
> changes, covoar needs to be adjusted.

This means fragile.

> The ignored record message I saw is in the code that reads Couverture
> trace records. The info in the record appears to be inconsistent with the
> code to which it has been matched.

Sorry, I do not understand why this difference is happening? I understand it is
object files vs executable, what I do not understand is why the object files are
being referenced, why not just use the executable?

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Coverage analysis update

[Chris Johns]

On 07/07/2017 08:11, Joel Sherrill wrote:
> 
> The rtems tester tries to run the tests in parallel. I wonder if this trips
> some conflict which we are not aware of.

Are the file generated cleanly separated? If you are rewriting files with qemu
and running covoar at the same time on the same files it may explain things.

> Chris.. Are there arguments to make it single-thread the qemu invocations
> as a test?

https://docs.rtems.org/branches/master/user/tools/index.html#command-line-help

--jobs ?

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Coverage analysis update

[Chris Johns]

On 07/07/2017 00:34, Joel Sherrill wrote:
> On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell  > wrote:
> >> When a test run is successful it will also finish with a lot of these 
> messages:
> Also I wanted to mention, there is quite a bit of work from the 2
> previous students on the integration of the coverage work into RTEMS
> Tester, would you be interested in starting the merging process of
> this. It might take a while to get everything reviewed and make any
> necessary changes. At this point it is working. its just fixing bugs,
> which I'm working on at the moment.
> 
> Chris.. I suggested this. It seems the basic flow and integration of
> coverage into the tester is good enough to review and merge. 
> 

Is there a place where this work is merged into something that can be reviewed?

Is there an explanation of the procedure being implemented? The last I could
find is https://lists.rtems.org/pipermail/devel/2014-May/006692.html, search for
"The procedure is as follows ...".

For merging I am interested in the merging of what may be happening externally
in scripts to use the functionality provided in the RTEMS Toolkit, for example
symbols.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Coverage analysis update

[Joel Sherrill]

On Thu, Jul 6, 2017 at 4:14 PM, Cillian O'Donnell 
wrote:

> On 6 July 2017 at 19:14, Joel Sherrill  wrote:
> >
> >
> > On Thu, Jul 6, 2017 at 11:11 AM, Cillian O'Donnell <
> cpodonne...@gmail.com>
> > wrote:
> >>
> >> On 6 July 2017 at 15:34, Joel Sherrill  wrote:
> >> >
> >> >
> >> > On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell
> >> > 
> >> > wrote:
> >> >>
> >> >> On 6 July 2017 at 08:05, Chris Johns  wrote:
> >> >> > On 05/07/2017 22:46, Cillian O'Donnell wrote:
> >> >> >>
> >> >> >> Now the thing is, I might have to run RTEMS Tester about 3 times
> for
> >> >> >> the full testsuite before I can get the report, as the covoar runs
> >> >> >> will often hang on an arbitrary method and have to be manually
> >> >> >> exited
> >> >> >> and re-run.
> >> >> >
> >> >> > How is covoar being run? Is it being run by the 'rtems-test'
> command?
> >> >>
> >> >> Yes covoar runs after the testsuite if a --coverage flag is included
> >> >> with rtems-test command.
> >> >> >
> >> >> >> The point that this happens is not repeatable and there's
> >> >> >> no error message to go on.
> >> >> >
> >> >> > Is it the covoar process that is stopping? Is it the process idle
> or
> >> >> > using 100%?
> >> >>
> >> >> Covoar locks up and becomes idle. Joel mentioned in the IRC meeting
> >> >> he's seen it before and I'll need to track down the exe causing it
> and
> >> >> debug it to get rid of this
> >> >
> >> >
> >> > There must be a misunderstanding. I have never seen covoar lock up
> >> > like this.
> >>
> >> Oh sorry, you must of just meant debugging covoar in general, picked
> >> it up wrong.
> >> >
> >> > It will ignore records when it thinks things are inconsistent. This
> can
> >> > occur
> >> > when a method appears in two different executables and has different
> >> > sizes. The cause of this is usually padding at the end of the method
> so
> >> > the subsequent method in memory starts on a nice cache-line alignment.
> >> > The code tries to recognize the nops/padding at the end and ignore
> them.
> >> > When the padding inserted by ld changes or the objdump output being
> >> > parsed changes, covoar needs to be adjusted.
> >> >
> >> > The ignored record message I saw is in the code that reads Couverture
> >> > trace records. The info in the record appears to be inconsistent with
> >> > the
> >> > code to which it has been matched. This could be because the trace
> >> > record format has been changed or a simple bug.
> >> >
> >> > From what I have seen, it is likely that it is primarily bugs in input
> >> > processing.
> >> >
> >> > But the code to do the "big union" of the trace data has to iterate
> over
> >> > all tests and all coverage blocks. It can take a while to run. I guess
> >> > there could be an infinite loop in it.  I always debugged it by having
> >> > a test case I could run in gdb.
> >> >
> >> >>
> >> >> >
> >> >> >> The only common theme is the methods will
> >> >> >> often be called thread...something. Even more rarely it will stop
> >> >> >> with
> >> >> >> the message 'resource temporarily unavailable'. I'm wondering is
> >> >> >> this
> >> >> >> normal behaviour for covoar on a long test-run? This never happens
> >> >> >> when running just the samples.
> >> >> >
> >> >> > It is not normal behavior.
> >> >
> >> >
> >> > +1
> >> >
> >> > Is Linux or covoar printing the message? It holds all data in memory
> >> > while
> >> > doing the analysis. Is it consuming a lot of RAM?
> >>
> >> I done two rtems-test coverage runs there while monitoring 'top'
> >> output. The first run is successful and coverage  report is generated
> >> and the second run produced IO Error: Resource temporarily unavailable
> >> on the second. Both times theres about 3.5-4GB RAM free and CPU runs
> >> at 70-80% so nothing seems to be strained there.
> >> Timecounter_Tick_simple is being checked before it happens. The output
> >> is
> >>
> >> 4000c9d8:0a 80 00 03 bcs  4000c9e4 <_Timecounter_Tick_simple+
> 0x78>
> >> 4000c9dc:84 01 20 01 add  %g4, 1, %g2
> >>
> >> bt = th->th_offset;
> >> 4000c9e0:84 10 00 04 mov  %g4, %g2
> >> _bt->sec += _bt2->sec;
> >> 4000c9e4:fa 06 e0 a0 ld  [ %i3 + 0xa0 ], %i5
> >> 4000c9e8:84 00 80 1d add  %g2, %i5, %g2
> >> _tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32)) >>
> >> 32;
> >> 4000c9ec:39 00 03 d0 sethi  %hi(0xf4000), %i4
> >> _ts->tv_nsec = ((uint64_t)10 *
> >> 4000c9f0:3b 0e e6 b2 sethi  %hi(0x3b9ac800), %i5
> >> _tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32)) >>
> >> 32;
> >> 4000c9f4:b8 17 22 40 or  %i4, 0x240, %i4
> >> _ts->tv_nsec = ((uint64_t)10 *
> >> 4000c9f8:ba 17 62 00 or  %i5, 0x200, %i5
> >> _tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32))
> >> >> 32;Exception in thread _stderr[c]:
> >> Traceback (most recent call last):

Re: Coverage analysis update

[Cillian O'Donnell]

On 6 July 2017 at 19:14, Joel Sherrill  wrote:
>
>
> On Thu, Jul 6, 2017 at 11:11 AM, Cillian O'Donnell 
> wrote:
>>
>> On 6 July 2017 at 15:34, Joel Sherrill  wrote:
>> >
>> >
>> > On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell
>> > 
>> > wrote:
>> >>
>> >> On 6 July 2017 at 08:05, Chris Johns  wrote:
>> >> > On 05/07/2017 22:46, Cillian O'Donnell wrote:
>> >> >>
>> >> >> Now the thing is, I might have to run RTEMS Tester about 3 times for
>> >> >> the full testsuite before I can get the report, as the covoar runs
>> >> >> will often hang on an arbitrary method and have to be manually
>> >> >> exited
>> >> >> and re-run.
>> >> >
>> >> > How is covoar being run? Is it being run by the 'rtems-test' command?
>> >>
>> >> Yes covoar runs after the testsuite if a --coverage flag is included
>> >> with rtems-test command.
>> >> >
>> >> >> The point that this happens is not repeatable and there's
>> >> >> no error message to go on.
>> >> >
>> >> > Is it the covoar process that is stopping? Is it the process idle or
>> >> > using 100%?
>> >>
>> >> Covoar locks up and becomes idle. Joel mentioned in the IRC meeting
>> >> he's seen it before and I'll need to track down the exe causing it and
>> >> debug it to get rid of this
>> >
>> >
>> > There must be a misunderstanding. I have never seen covoar lock up
>> > like this.
>>
>> Oh sorry, you must of just meant debugging covoar in general, picked
>> it up wrong.
>> >
>> > It will ignore records when it thinks things are inconsistent. This can
>> > occur
>> > when a method appears in two different executables and has different
>> > sizes. The cause of this is usually padding at the end of the method so
>> > the subsequent method in memory starts on a nice cache-line alignment.
>> > The code tries to recognize the nops/padding at the end and ignore them.
>> > When the padding inserted by ld changes or the objdump output being
>> > parsed changes, covoar needs to be adjusted.
>> >
>> > The ignored record message I saw is in the code that reads Couverture
>> > trace records. The info in the record appears to be inconsistent with
>> > the
>> > code to which it has been matched. This could be because the trace
>> > record format has been changed or a simple bug.
>> >
>> > From what I have seen, it is likely that it is primarily bugs in input
>> > processing.
>> >
>> > But the code to do the "big union" of the trace data has to iterate over
>> > all tests and all coverage blocks. It can take a while to run. I guess
>> > there could be an infinite loop in it.  I always debugged it by having
>> > a test case I could run in gdb.
>> >
>> >>
>> >> >
>> >> >> The only common theme is the methods will
>> >> >> often be called thread...something. Even more rarely it will stop
>> >> >> with
>> >> >> the message 'resource temporarily unavailable'. I'm wondering is
>> >> >> this
>> >> >> normal behaviour for covoar on a long test-run? This never happens
>> >> >> when running just the samples.
>> >> >
>> >> > It is not normal behavior.
>> >
>> >
>> > +1
>> >
>> > Is Linux or covoar printing the message? It holds all data in memory
>> > while
>> > doing the analysis. Is it consuming a lot of RAM?
>>
>> I done two rtems-test coverage runs there while monitoring 'top'
>> output. The first run is successful and coverage  report is generated
>> and the second run produced IO Error: Resource temporarily unavailable
>> on the second. Both times theres about 3.5-4GB RAM free and CPU runs
>> at 70-80% so nothing seems to be strained there.
>> Timecounter_Tick_simple is being checked before it happens. The output
>> is
>>
>> 4000c9d8:0a 80 00 03 bcs  4000c9e4 <_Timecounter_Tick_simple+0x78>
>> 4000c9dc:84 01 20 01 add  %g4, 1, %g2
>>
>> bt = th->th_offset;
>> 4000c9e0:84 10 00 04 mov  %g4, %g2
>> _bt->sec += _bt2->sec;
>> 4000c9e4:fa 06 e0 a0 ld  [ %i3 + 0xa0 ], %i5
>> 4000c9e8:84 00 80 1d add  %g2, %i5, %g2
>> _tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32)) >>
>> 32;
>> 4000c9ec:39 00 03 d0 sethi  %hi(0xf4000), %i4
>> _ts->tv_nsec = ((uint64_t)10 *
>> 4000c9f0:3b 0e e6 b2 sethi  %hi(0x3b9ac800), %i5
>> _tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32)) >>
>> 32;
>> 4000c9f4:b8 17 22 40 or  %i4, 0x240, %i4
>> _ts->tv_nsec = ((uint64_t)10 *
>> 4000c9f8:ba 17 62 00 or  %i5, 0x200, %i5
>> _tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32))
>> >> 32;Exception in thread _stderr[c]:
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
>> self.run()
>>   File "/usr/lib/python2.7/threading.py", line 754, in run
>> self.__target(*self.__args, **self.__kwargs)
>>   File
>> "/home/cpod/development/rtems/test/rtems-tools/rtemstoolkit/execute.py",
>> line 206, in _readthread
>> 

Re: Coverage analysis update

[Joel Sherrill]

On Thu, Jul 6, 2017 at 11:11 AM, Cillian O'Donnell 
wrote:

> On 6 July 2017 at 15:34, Joel Sherrill  wrote:
> >
> >
> > On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell  >
> > wrote:
> >>
> >> On 6 July 2017 at 08:05, Chris Johns  wrote:
> >> > On 05/07/2017 22:46, Cillian O'Donnell wrote:
> >> >>
> >> >> Now the thing is, I might have to run RTEMS Tester about 3 times for
> >> >> the full testsuite before I can get the report, as the covoar runs
> >> >> will often hang on an arbitrary method and have to be manually exited
> >> >> and re-run.
> >> >
> >> > How is covoar being run? Is it being run by the 'rtems-test' command?
> >>
> >> Yes covoar runs after the testsuite if a --coverage flag is included
> >> with rtems-test command.
> >> >
> >> >> The point that this happens is not repeatable and there's
> >> >> no error message to go on.
> >> >
> >> > Is it the covoar process that is stopping? Is it the process idle or
> >> > using 100%?
> >>
> >> Covoar locks up and becomes idle. Joel mentioned in the IRC meeting
> >> he's seen it before and I'll need to track down the exe causing it and
> >> debug it to get rid of this
> >
> >
> > There must be a misunderstanding. I have never seen covoar lock up
> > like this.
>
> Oh sorry, you must of just meant debugging covoar in general, picked
> it up wrong.
> >
> > It will ignore records when it thinks things are inconsistent. This can
> > occur
> > when a method appears in two different executables and has different
> > sizes. The cause of this is usually padding at the end of the method so
> > the subsequent method in memory starts on a nice cache-line alignment.
> > The code tries to recognize the nops/padding at the end and ignore them.
> > When the padding inserted by ld changes or the objdump output being
> > parsed changes, covoar needs to be adjusted.
> >
> > The ignored record message I saw is in the code that reads Couverture
> > trace records. The info in the record appears to be inconsistent with the
> > code to which it has been matched. This could be because the trace
> > record format has been changed or a simple bug.
> >
> > From what I have seen, it is likely that it is primarily bugs in input
> > processing.
> >
> > But the code to do the "big union" of the trace data has to iterate over
> > all tests and all coverage blocks. It can take a while to run. I guess
> > there could be an infinite loop in it.  I always debugged it by having
> > a test case I could run in gdb.
> >
> >>
> >> >
> >> >> The only common theme is the methods will
> >> >> often be called thread...something. Even more rarely it will stop
> with
> >> >> the message 'resource temporarily unavailable'. I'm wondering is this
> >> >> normal behaviour for covoar on a long test-run? This never happens
> >> >> when running just the samples.
> >> >
> >> > It is not normal behavior.
> >
> >
> > +1
> >
> > Is Linux or covoar printing the message? It holds all data in memory
> while
> > doing the analysis. Is it consuming a lot of RAM?
>
> I done two rtems-test coverage runs there while monitoring 'top'
> output. The first run is successful and coverage  report is generated
> and the second run produced IO Error: Resource temporarily unavailable
> on the second. Both times theres about 3.5-4GB RAM free and CPU runs
> at 70-80% so nothing seems to be strained there.
> Timecounter_Tick_simple is being checked before it happens. The output
> is
>
> 4000c9d8:0a 80 00 03 bcs  4000c9e4 <_Timecounter_Tick_simple+0x78>
> 4000c9dc:84 01 20 01 add  %g4, 1, %g2
>
> bt = th->th_offset;
> 4000c9e0:84 10 00 04 mov  %g4, %g2
> _bt->sec += _bt2->sec;
> 4000c9e4:fa 06 e0 a0 ld  [ %i3 + 0xa0 ], %i5
> 4000c9e8:84 00 80 1d add  %g2, %i5, %g2
> _tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32)) >> 32;
> 4000c9ec:39 00 03 d0 sethi  %hi(0xf4000), %i4
> _ts->tv_nsec = ((uint64_t)10 *
> 4000c9f0:3b 0e e6 b2 sethi  %hi(0x3b9ac800), %i5
> _tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32)) >> 32;
> 4000c9f4:b8 17 22 40 or  %i4, 0x240, %i4
> _ts->tv_nsec = ((uint64_t)10 *
> 4000c9f8:ba 17 62 00 or  %i5, 0x200, %i5
> _tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32))
> >> 32;Exception in thread _stderr[c]:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
> self.run()
>   File "/usr/lib/python2.7/threading.py", line 754, in run
> self.__target(*self.__args, **self.__kwargs)
>   File "/home/cpod/development/rtems/test/rtems-tools/rtemstoolkit/
> execute.py",
> line 206, in _readthread
> _output_line(line, exe, prefix, out, count)
>   File "/home/cpod/development/rtems/test/rtems-tools/rtemstoolkit/
> execute.py",
> line 184, in _output_line
> out(prefix + line)
>   File 

Re: Coverage analysis update

[Cillian O'Donnell]

On 6 July 2017 at 15:34, Joel Sherrill  wrote:
>
>
> On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell 
> wrote:
>>
>> On 6 July 2017 at 08:05, Chris Johns  wrote:
>> > On 05/07/2017 22:46, Cillian O'Donnell wrote:
>> >>
>> >> Now the thing is, I might have to run RTEMS Tester about 3 times for
>> >> the full testsuite before I can get the report, as the covoar runs
>> >> will often hang on an arbitrary method and have to be manually exited
>> >> and re-run.
>> >
>> > How is covoar being run? Is it being run by the 'rtems-test' command?
>>
>> Yes covoar runs after the testsuite if a --coverage flag is included
>> with rtems-test command.
>> >
>> >> The point that this happens is not repeatable and there's
>> >> no error message to go on.
>> >
>> > Is it the covoar process that is stopping? Is it the process idle or
>> > using 100%?
>>
>> Covoar locks up and becomes idle. Joel mentioned in the IRC meeting
>> he's seen it before and I'll need to track down the exe causing it and
>> debug it to get rid of this
>
>
> There must be a misunderstanding. I have never seen covoar lock up
> like this.

Oh sorry, you must of just meant debugging covoar in general, picked
it up wrong.
>
> It will ignore records when it thinks things are inconsistent. This can
> occur
> when a method appears in two different executables and has different
> sizes. The cause of this is usually padding at the end of the method so
> the subsequent method in memory starts on a nice cache-line alignment.
> The code tries to recognize the nops/padding at the end and ignore them.
> When the padding inserted by ld changes or the objdump output being
> parsed changes, covoar needs to be adjusted.
>
> The ignored record message I saw is in the code that reads Couverture
> trace records. The info in the record appears to be inconsistent with the
> code to which it has been matched. This could be because the trace
> record format has been changed or a simple bug.
>
> From what I have seen, it is likely that it is primarily bugs in input
> processing.
>
> But the code to do the "big union" of the trace data has to iterate over
> all tests and all coverage blocks. It can take a while to run. I guess
> there could be an infinite loop in it.  I always debugged it by having
> a test case I could run in gdb.
>
>>
>> >
>> >> The only common theme is the methods will
>> >> often be called thread...something. Even more rarely it will stop with
>> >> the message 'resource temporarily unavailable'. I'm wondering is this
>> >> normal behaviour for covoar on a long test-run? This never happens
>> >> when running just the samples.
>> >
>> > It is not normal behavior.
>
>
> +1
>
> Is Linux or covoar printing the message? It holds all data in memory while
> doing the analysis. Is it consuming a lot of RAM?

I done two rtems-test coverage runs there while monitoring 'top'
output. The first run is successful and coverage  report is generated
and the second run produced IO Error: Resource temporarily unavailable
on the second. Both times theres about 3.5-4GB RAM free and CPU runs
at 70-80% so nothing seems to be strained there.
Timecounter_Tick_simple is being checked before it happens. The output
is

4000c9d8:0a 80 00 03 bcs  4000c9e4 <_Timecounter_Tick_simple+0x78>
4000c9dc:84 01 20 01 add  %g4, 1, %g2

bt = th->th_offset;
4000c9e0:84 10 00 04 mov  %g4, %g2
_bt->sec += _bt2->sec;
4000c9e4:fa 06 e0 a0 ld  [ %i3 + 0xa0 ], %i5
4000c9e8:84 00 80 1d add  %g2, %i5, %g2
_tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32)) >> 32;
4000c9ec:39 00 03 d0 sethi  %hi(0xf4000), %i4
_ts->tv_nsec = ((uint64_t)10 *
4000c9f0:3b 0e e6 b2 sethi  %hi(0x3b9ac800), %i5
_tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32)) >> 32;
4000c9f4:b8 17 22 40 or  %i4, 0x240, %i4
_ts->tv_nsec = ((uint64_t)10 *
4000c9f8:ba 17 62 00 or  %i5, 0x200, %i5
_tv->tv_usec = ((uint64_t)100 * (uint32_t)(_bt->frac >> 32))
>> 32;Exception in thread _stderr[c]:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
  File "/home/cpod/development/rtems/test/rtems-tools/rtemstoolkit/execute.py",
line 206, in _readthread
_output_line(line, exe, prefix, out, count)
  File "/home/cpod/development/rtems/test/rtems-tools/rtemstoolkit/execute.py",
line 184, in _output_line
out(prefix + line)
  File "/home/cpod/development/rtems/test/rtems-tools/tester/rt/coverage.py",
line 387, in output_handler
log.notice(text, stdout_only = False)
  File "/home/cpod/development/rtems/test/rtems-tools/rtemstoolkit/log.py",
line 104, in notice
print(l)
IOError: [Errno 11] Resource temporarily unavailable

>
>>
>> >> When a test run is successful it 

Re: Coverage analysis update

[Joel Sherrill]

On Thu, Jul 6, 2017 at 5:53 AM, Cillian O'Donnell 
wrote:

> On 6 July 2017 at 08:05, Chris Johns  wrote:
> > On 05/07/2017 22:46, Cillian O'Donnell wrote:
> >>
> >> Now the thing is, I might have to run RTEMS Tester about 3 times for
> >> the full testsuite before I can get the report, as the covoar runs
> >> will often hang on an arbitrary method and have to be manually exited
> >> and re-run.
> >
> > How is covoar being run? Is it being run by the 'rtems-test' command?
>
> Yes covoar runs after the testsuite if a --coverage flag is included
> with rtems-test command.
> >
> >> The point that this happens is not repeatable and there's
> >> no error message to go on.
> >
> > Is it the covoar process that is stopping? Is it the process idle or
> using 100%?
>
> Covoar locks up and becomes idle. Joel mentioned in the IRC meeting
> he's seen it before and I'll need to track down the exe causing it and
> debug it to get rid of this
>

There must be a misunderstanding. I have never seen covoar lock up
like this.

It will ignore records when it thinks things are inconsistent. This can
occur
when a method appears in two different executables and has different
sizes. The cause of this is usually padding at the end of the method so
the subsequent method in memory starts on a nice cache-line alignment.
The code tries to recognize the nops/padding at the end and ignore them.
When the padding inserted by ld changes or the objdump output being
parsed changes, covoar needs to be adjusted.

The ignored record message I saw is in the code that reads Couverture
trace records. The info in the record appears to be inconsistent with the
code to which it has been matched. This could be because the trace
record format has been changed or a simple bug.

>From what I have seen, it is likely that it is primarily bugs in input
processing.

But the code to do the "big union" of the trace data has to iterate over
all tests and all coverage blocks. It can take a while to run. I guess
there could be an infinite loop in it.  I always debugged it by having
a test case I could run in gdb.


> >
> >> The only common theme is the methods will
> >> often be called thread...something. Even more rarely it will stop with
> >> the message 'resource temporarily unavailable'. I'm wondering is this
> >> normal behaviour for covoar on a long test-run? This never happens
> >> when running just the samples.
> >
> > It is not normal behavior.
>

+1

Is Linux or covoar printing the message? It holds all data in memory while
doing the analysis. Is it consuming a lot of RAM?


> >> When a test run is successful it will also finish with a lot of these
> messages:
> >>
> >> *** Trace block is inconsistent with coverage map
> >> *** Trace block (0x4000c4fc - 0x4000c51f) for 36 bytes
> >> *** Coverage map /home/cpod/coverage_test/test/
> coverage/unlimited.exe.cov
> >> INFO: DesiredSymbols::mergeCoverageMap - Unable to merge coverage map
> >> for _Thread_queue_Extract_with_proxy because the sizes are different
> >
> > I wonder what sizes are different?
>
> For this one, Joel mentioned it could be something to do with nops
> being added or space not matching and to take a look at _Thread_
> queue_Extract_with_proxy  in the objdump and see if I can dig up
> anything from that. That message appears for _Thread_
> queue_Extract_with_proxy  on all tests from base_sp.exe to
> unlimited.exe when running the full testsuite and on a few instances
> it also mentions unable to merge coverage map for TOD_TICKS_PER_SECOND
> for the same reason. The mismatch is always 36 bytes.
>

You need to find the two executables it differs on and see why the size
is different. It processes them from first to last on the command line so
find the first one that has the symbol and compare the dump to that
one it is flagging as different.

>
> Also I wanted to mention, there is quite a bit of work from the 2
> previous students on the integration of the coverage work into RTEMS
> Tester, would you be interested in starting the merging process of
> this. It might take a while to get everything reviewed and make any
> necessary changes. At this point it is working. its just fixing bugs,
> which I'm working on at the moment.
>

Chris.. I suggested this. It seems the basic flow and integration of
coverage into the tester is good enough to review and merge.

--joel


>
> Thanks,
>
> Cillian.
> >
> > Chris
> ___
> devel mailing list
> devel@rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Coverage analysis update

[Chris Johns]

On 05/07/2017 22:46, Cillian O'Donnell wrote:
> 
> Now the thing is, I might have to run RTEMS Tester about 3 times for
> the full testsuite before I can get the report, as the covoar runs
> will often hang on an arbitrary method and have to be manually exited
> and re-run. 

How is covoar being run? Is it being run by the 'rtems-test' command?

> The point that this happens is not repeatable and there's
> no error message to go on.

Is it the covoar process that is stopping? Is it the process idle or using 100%?

> The only common theme is the methods will
> often be called thread...something. Even more rarely it will stop with
> the message 'resource temporarily unavailable'. I'm wondering is this
> normal behaviour for covoar on a long test-run? This never happens
> when running just the samples.

It is not normal behavior.

> 
> When a test run is successful it will also finish with a lot of these 
> messages:
> 
> *** Trace block is inconsistent with coverage map
> *** Trace block (0x4000c4fc - 0x4000c51f) for 36 bytes
> *** Coverage map /home/cpod/coverage_test/test/coverage/unlimited.exe.cov
> INFO: DesiredSymbols::mergeCoverageMap - Unable to merge coverage map
> for _Thread_queue_Extract_with_proxy because the sizes are different

I wonder what sizes are different?

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel