Call for Papers
First Workshop on “ROC Analysis in AI”, ROCAI2004
http://www.dsic.upv.es/~flip/ROCAI2004/ Valencia, Spain, 22-23 August, 2004
Submission Deadline: 15 April, 2004.to be held within ECAI’2004, http://www.dsic.upv.es/ecai2004/ the European Conference on Artificial Intelligence.
-------- Overview --------
Receiver Operating Characteristic Analysis (ROC Analysis) is related in a direct and natural way to cost/benefit analysis of diagnostic decision making. Widely used in medicine for many decades, it has been introduced relatively recently in some areas of artificial intelligence: machine learning, multiagent systems, intelligent decision support and expert systems. In this context, ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution. Furthermore, the Area Under the ROC Curve (AUC) has been shown to be a better evaluation measure than accuracy in contexts with variable misclassification costs and/or imbalanced datasets. AUC is also the standard measure when using classifiers to rank examples, and, hence, is used in applications where ranking is crucial, such as campaign design, model combination, collaboration strategies, and co-learning.
Nevertheless, there are many open questions and some limitations that hamper a broader use and applicability of ROC analysis. Its connections with other evaluation measures is not yet clarified completely, its incorporation in decision support and expert systems technology just envisaged, its use for improving the decisions of (communities of) intelligent agents unexplored, and its use in data mining still below its full potential. Among the limitations of ROC analysis, an important one, despite some recent progress, is its possible but difficult extension for more than two classes.
One of the main goals of the workshop is to foster the cross-fertilisation of ideas and applications with related areas in artificial intelligence. Consequently, the presentations and discussions will be open to a broad list of topics (not exhaustive):
* Adaptation of classical learning methods for improving AUC instead of accuracy. * Agent selection/ranking in multiagent systems. * Alternative ROC representations. * Alternatives to AUC measures (such as AUC*). * Analysis of the performance of learners or decision systems based on AUC. * Applications of ROC Analysis in Expert Systems. * Classifier Evaluation. * Co-learning (collaborative learning) and ROC Analysis. * Consensus and Collaborative/Distributed Decision Making. * Constraint Satisfaction Methods for ROC Analysis. * Cost-sensitive Learning. * Decision Networks and Probabilistic Reasoning with ROC Analysis. * Expert Systems, Knowledge Bases and ROC Analysis. * Inductive Logic Programming and ROC Analysis. * Multi-class ROC on top of binary ROC. * Oversampling and ROC analysis. * Precision & Recall measures in Information Retrieval. * Ranking Actions. Applications in Planning and Robotics with variable contexts. * Reinforcement Learning and ROC analysis. * ROC Analysis for Model Building and Modification. * ROC Analysis for Descriptive Data Mining (association rules, subgroup discovery). * Soft classifiers and Probability Estimators. * Software Packages and Efficient Implementations (Convex Hull). * Use of ROC analysis and measures as fitness and selection criteria in evolutionary techniques. * Working with Imbalanced datasets.
----------------------- Details of the workshop -----------------------
This is a one-day workshop, with sessions consisting of short paper presentations, devoting an important share of time to informal discussion and interaction between the participants. The workshop will be closed with an open discussion about more promising open problems and research areas of ROC analysis, continuation of the workshop, future related events, etc.
Papers will be selected by the program committee according to the quality of the submission and its relevance to the workshop topics. All accepted papers will be gathered in printed form and distributed to registered attendees as workshop notes. The publication of a selected set of papers for a special volume or a journal issue is considered, but this will depend on the success and overall results of the workshop.
--------------- Important dates ---------------
* Deadline for the workshop papers: 15 April 2004 * Notification of acceptance: 15 May 2004 * Camera Ready copies: 1 June 2004 * Workshop dates: 22 or 23 August 2004 (soon to be determined, depending on the ECAI organising committee)
-------------------------- Workshop Program Committee --------------------------
* Hendrik Blockeel, K.U.Leuven, Belgium. * Stephan Dreiseitl FHS Hagenberg, Austria. * Tom Fawcett, HP Labs, Palo Alto, CA, USA. * Cèsar Ferri, Technical University of Valencia, Spain. * Peter Flach, University of Bristol, UK. * Johannes Fürnkranz, TU Darmstadt, Germany. * José Hernández-Orallo, Technical University of Valencia, Spain. * Nicolas Lachiche, University of Strasbourg, France. * Charles Ling, University of Western Ontario, Canada * Maarten van Someren, University of Amsterdam, The Netherlands. * Francesco Tortorella, University of Cassino, Italy.
----------------------------- Workshop Organizing Committee -----------------------------
* C. Ferri, Technical University of Valencia, Spain. * P. Flach, University of Bristol, UK. * J. Hernández-Orallo, T. University of Valencia, Spain (contact person). * N. Lachiche, University of Strasbourg, France.
--------------------- Submission Guidelines ---------------------
Potential participants are invited to submit short papers (no longer than 10 pages), which may be in the form of a technical paper, a position paper (e.g. highlighting open problems or new applications of ROC analysis), an overview of their research or a software demonstration.
Authors should submit their papers electronically (PDF or PS format) to the contact person ([EMAIL PROTECTED]). It is recommended to submit papers using the final camera-ready ECAI 2004 conference paper style (http://www.dsic.upv.es/ecai2004/cfp/style/style.html) including author names.
Those wishing to attend the meeting without submitting a paper should send a short statement of interest to the contact person describing their work or interest in the area.
- -------------------------------------------------------------------
Cèsar Ferri http://www.dsic.upv.es/~cferri
Phone: +34 96 3877007 Ext.: 83505
email: [EMAIL PROTECTED]
Dpt. Sistemes Informatics i Computacio http://www.dsic.upv.es
Universitat Politecnica de Valencia http://www.upv.es
Phone: +34 96 3877350 Fax: +34 96 3877359
Address: Cami de Vera s/n E-46022 Valencia (Spain)
------------------------------------------------------------------ -
