Apologies for cross-posting.
I would like to invite you to participate in the RERS Challenge 2013
http://rers-challenge.org/
which is part of ASE 2013 in Palo ALto (CA)
http://ase2013.org/
The RERS challenge series is unique in its competition modus which allows
participation of groups with very different profile, ranging form program
analysis and (interactive) verification, over model checking to runtime
verification, (model-based) testing, active learning (test-based
modelling), and passive unsupervised learning.
For machine learning, an interesting property of the RERS challenge is that
it aims to use models not only for prediction, but to draw conclusions on
the system level. Tools are made available for determining such properties
given a learned state machine model.
The challenge is in particular interesting to researchers in passive and
active state machine learning, grammatical inference, process mining, but
also techniques for learning Hidden Markov models or simpler methods such
an N-grams can be applied. For machine learning approaches, there will be
two objectives:
- Traditional evaluation of passive learning based on classification
accuracy on an unseen test set. This is similar to a one-class
classification problem for sequence data (execution traces).
- Answering logical queries on the language from which the traces are
sampled. The answers to the logical queries from the learned models will be
compared to the answers of program analysis.
More info is given below and available on the website.
=======================================================
The RERS challenge series aims at a systematic investigation, evaluation,
comparison, combination and improvement of any kind of methods for the
analysis and validation of reactive systems, be they static, dynamic, black
box or white box. Its second edition, the (RERS Challenge 2012), focused on
ECA systems, a popular class of reactive systems, which comprises in
particular Web services, decision support systems, and programmable logical
controllers (PLCs). Besides their industrial relevance, ECA systems where
chosen as they are on the one hand fully 'white-box': the full Java/C code
will be available, but one the other hand have a black-box character: being
simply one huge loop of guarded commands, the ECA code structure
essentially reveals nothing about the implemented functionality. This
property was meant to also address competitors who base their validation on
execution rather than source code analysis.
The RERS Challenge 2013 builds on last year's experience. It involves more
complex code/data structures in order to attract source code analyzers, and
it explicitly addresses execution-based analyses by providing black box and
grey box scenarios. In particular the latter scenarios are challenging as
they profit most from the combination of source code and execution-based
analyses. We therefore hope to encourage people working on areas as diverse
as
program analysis and verification,
symbolic execution,
software model checking,
statistical model checking,
model-based testing,
inference of invariants,
automata learning,
run-time verification,
monitoring.
to not only apply there 'home' methods, but to investigate how their
methods can be improved by combining them with others.
Challenge Problems:
The RERS Challenge 2013 will provide a wealth of Benchmark problems of
increasing complexity, the more involved of which will probably be beyond
any individual state-of-the-art method or tool. A large set of Benchmarks
will be synthesized to exhibit chosen properties, and then enhanced in an
automated process to cover dedicated dimensions of difficulty, including:
conceptual complexity of the exhibited properties (reachability,
safety, liveness),
size of the systems (from a few hundred lines of code to millions of
them), and
language features (arrays, indirect addressing, floating point
arithmetics, virtual method calls, recursion).
Automatically generated problems will be presented
white box: i.e. in Java and C,
black box: as executables, not meant to be textually analyzed,
grey box: a mixture thereof.
The challenge rules are essentially free style in order include as many
participants as possible. They have a numeric part, which allows for a
clear ranking in terms of number of correctly answered questions, and a
textual part, where competitors are supposed to describe the approach. This
part will be evaluated by the RERS committee. Details about the ranking or
evaluation method can be found here.
There are five price categories: white box, black box, grey box, overall
and for the best approach taken (which must not necessarily have scored
highest). In addition we will distribute achievement certificates for
solutions passing a specific threshold.
Rewards
There will be an overall price and price for each kind, as well as a price
for the best conceptual contribution in form of a gift certificate for
Springer books sponsored by Springer. In addition there will be
certificates for the first three in each category, and achievement
certificates for every team passing the required threshold.
The teams with the best solutions in their categories will be invited for a
STTT Special Section summarizing the results of the challenge, and, in
particular, presenting the most advanced solutions.
_______________________________________________
uai mailing list
[email protected]
https://secure.engr.oregonstate.edu/mailman/listinfo/uai
