On 7/29/2013 3:09 PM, Krzysztof Mięsowicz wrote:
Thank you for all replies very much.
I read about libfiu and it seems really interesting and I believe it
can be quite easy applied to RTEMS testing. I hope so :)
The process of using preloaded libraries to intercept calls and
break them may not work on RTEMS at all. Plus the idea of
running a process under another process is foreign since we
only have one process. I don't have a good feeling on libfiu
for RTEMS on an admittedly cursory examination.
API parameter fuzzing was done in the report I posted.
I did a quick google and found http://peachfuzzer.com/
which looks like a FLOSS software package to what was
done in that report. The Classic APIs were described in
terms of their data types. For each data type, "interesting"
values were defined. Say NULL for argument pointers,
or 0xfffffffc and 0xffffffff for the start of a region or
partition.
As I recall, the challenge is that the way RTEMS APIs
work, you often need a valid object ID or N-1 parameters
to be valid to get to the Nth parameter. So there was
some custom test setup to write to go along with the
automatically generated fuzzing code. Depending on the
fuzzing package, this may be handled better.
From my perspective, API parameter fuzzing was valuable
at the time. It turned up some odd bugs that wouldn't
likely have been caught otherwise.
With all that said, I don't know how much you can
accomplish in the time period. The goal would have
to be to provide the foundation for a permanent
addition to the test suite which can be added to.
You can't possibly finish all APIs so we would have to
define an interesting subset.
According to Joel's mail:
/Another area which we need help in is the coverage and testing
scripts. The coverage reporting really should be by directory or
capability. It is now reported in "lumps" that are too large.
Also rewriting as much of this as possible in Python including the
sim-scripts is highly desirable./
Do you mean that coverage reporting framework should be rewrited and
reports should be generated for source directories instead of core and
developmental parts (as it is done now) ?
Yes. Having core is OK but developmental really sucks. When I
added filesystems, the overall percentage dropped significantly.
What we want to see from an analysis and marketing perspective
is that directory X is 99.9% but this new directory Y is at 50%
and needs testing attention.
As we move more into continuous integration testing, this
fine grained reporting will be even more valuable.
The scripts that drive this process are in shell. We are moving to using
Python as our host tools scripting language. So these should be
rewritten.
This project is more defined and has a short term completion.
And what particularly do you mean by "rewriting as much of THIS as
possible" - do you mean all coverage reporting scripts which is now
written as bash scripts? Do I understand you correctly?
Yes. There is "covoar" which is C++ but the contents of rtems-coverage
are pretty
much all shell scripts with some template files.
The rtems-testing git module has sim-scripts to run simulators and we want
to move them to Python as well. In the process we want to both them to
a single command rather than one per BSP. Invoked omething like
rtems-sim [args] BSP
And rtems-testing has other scripts.
Greetings,
Krzysiek
2013/7/29 Rempel, Cynthia <cynt6...@vandals.uidaho.edu
<mailto:cynt6...@vandals.uidaho.edu>>
Hi,
Did a quick write-up on the fault-injection open-project...
http://blitiri.com.ar/p/libfiu/ looks interesting... (the python
bindings have been Debian packaged)
it would probably need to be evaluated more thoroughly as part of
the project though...
Thanks,
Cindy
________________________________________
From: rtems-devel-boun...@rtems.org
<mailto:rtems-devel-boun...@rtems.org>
[rtems-devel-boun...@rtems.org
<mailto:rtems-devel-boun...@rtems.org>] on behalf of Rempel,
Cynthia [cynt6...@vandals.uidaho.edu
<mailto:cynt6...@vandals.uidaho.edu>]
Sent: Monday, July 29, 2013 9:06 AM
To: Joel Sherrill; Gedare Bloom
Cc: Krzysztof Mięsowicz; rtems-devel@rtems.org
<mailto:rtems-devel@rtems.org>
Subject: RE: ESA SOCIS - Fault injection tools topic
Hi,
Thanks for the link! This will be helpful with fleshing out the
project...
Yes, the initial idea was to get a tool the RTEMS development
community could use to help with certification efforts...
and having the certification report would be very helpful...
Given this information, we can start to write an open-project page
for this...
The criteria we typically use for evaluating other projects are:
1. Free and open source (usually modified BSD or special exception
GPLv2, but as this is testing GPL would be fine)
2. Works under Linux and Windows
3. Still under maintainence
Thanks!
Cindy
________________________________________
From: rtems-devel-boun...@rtems.org
<mailto:rtems-devel-boun...@rtems.org>
[rtems-devel-boun...@rtems.org
<mailto:rtems-devel-boun...@rtems.org>] on behalf of Joel Sherrill
[joel.sherr...@oarcorp.com]
Sent: Monday, July 29, 2013 8:23 AM
To: Gedare Bloom
Cc: Krzysztof Mięsowicz; rtems-devel@rtems.org
<mailto:rtems-devel@rtems.org>
Subject: Re: ESA SOCIS - Fault injection tools topic
If I forgot the link:
ftp://www.rtems.org/pub/rtems/esa_validation_report_450/
On 7/29/2013 9:45 AM, Gedare Bloom wrote:
> This seems like a good project. Ballista looks like it is "dead"
> upstream. You'll want to scope your project to decide what / how
much
> of RTEMS you can reasonably instrument with fault injection. I also
> wonder if it would be sensible to first start with a tool that can
> support test input "fuzzing", which is related to but somewhat
> different than fault injection. I think "fuzzing" is more
> appropriately related to test coverage /testing, whereas fault
> injection can include also non-valid inputs and faults, such as
> flipping bits in code or modifying values on-the-fly.
>
> On Mon, Jul 29, 2013 at 4:25 AM, Krzysztof Mięsowicz
> <krzysztof.miesow...@gmail.com
<mailto:krzysztof.miesow...@gmail.com>> wrote:
>> Hi!
>>
>> My name is Krzysiek. I participated in ESA SOCIS 2012 for RTEMS
and worked
>> on improving test coverage. I'm interested in participating in
this edition
>> of SOCIS again.
>>
>> I browse Open Projects page of RTEMS and I have question about
topic related
>> to fault injection tools. Have you any tips or suggestions to
this topic
>> (especially which tools are most desired or promising and how
this project
>> should look in details).
>>
>> I googled a bit about fault injection and free tools and find some:
>> Ballista, LFI, Fuzz, or JACA which potentially could be applied
to RTEMS
>> testing. But I'm looking forward to any tips or thoughts from
you and
>> additionaly is it possible to be selected again this year.
>>
>> Thanks in advance for your replies.
>> Krzysiek Mięsowicz
>>
>> _______________________________________________
>> rtems-devel mailing list
>> rtems-devel@rtems.org <mailto:rtems-devel@rtems.org>
>> http://www.rtems.org/mailman/listinfo/rtems-devel
>>
> _______________________________________________
> rtems-devel mailing list
> rtems-devel@rtems.org <mailto:rtems-devel@rtems.org>
> http://www.rtems.org/mailman/listinfo/rtems-devel
--
Joel Sherrill, Ph.D. Director of Research & Development
joel.sherr...@oarcorp.com On-Line Applications Research
Ask me about RTEMS: a free RTOS Huntsville AL 35805
Support Available (256) 722-9985 <tel:%28256%29%20722-9985>
_______________________________________________
rtems-devel mailing list
rtems-devel@rtems.org <mailto:rtems-devel@rtems.org>
http://www.rtems.org/mailman/listinfo/rtems-devel
_______________________________________________
rtems-devel mailing list
rtems-devel@rtems.org <mailto:rtems-devel@rtems.org>
http://www.rtems.org/mailman/listinfo/rtems-devel
--
Joel Sherrill, Ph.D. Director of Research & Development
joel.sherr...@oarcorp.com On-Line Applications Research
Ask me about RTEMS: a free RTOS Huntsville AL 35805
Support Available (256) 722-9985
_______________________________________________
rtems-devel mailing list
rtems-devel@rtems.org
http://www.rtems.org/mailman/listinfo/rtems-devel
