Author: pkluegl Date: Wed Jun 12 12:57:24 2013 New Revision: 1492177 URL: http://svn.apache.org/r1492177 Log: UIMA-2859 - added section about constraint driven evaluation
Added: uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/images/tools/ruta/workbench/cde/ uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/images/tools/ruta/workbench/cde/cde.png (with props) uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/tools.ruta.workbench.cde.xml Modified: uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/tools.ruta.workbench.xml Added: uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/images/tools/ruta/workbench/cde/cde.png URL: http://svn.apache.org/viewvc/uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/images/tools/ruta/workbench/cde/cde.png?rev=1492177&view=auto ============================================================================== Binary file - no diff available. Propchange: uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/images/tools/ruta/workbench/cde/cde.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/tools.ruta.workbench.cde.xml URL: http://svn.apache.org/viewvc/uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/tools.ruta.workbench.cde.xml?rev=1492177&view=auto ============================================================================== --- uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/tools.ruta.workbench.cde.xml (added) +++ uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/tools.ruta.workbench.cde.xml Wed Jun 12 12:57:24 2013 @@ -0,0 +1,90 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" +"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"[ +<!ENTITY imgroot "images/tools/ruta/workbench/" > +<!ENTITY % uimaents SYSTEM "../../target/docbook-shared/entities.ent" > +%uimaents; +]> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +<section id="section.tools.ruta.workbench.cde"> + <title>UIMA Ruta CDE perspective</title> + <para> + The UIMA Ruta CDE (Constraint-Driven Evaluation) provides some views and functionality to investigate and evaluate a set + of annotated documents using domain and background knowledge formalized as constraints and UIMA Ruta rules. + <xref linkend="figure.tools.ruta.workbench.cde"/> provides an overview of the perspective. Its views are described in the following. + </para> + <figure id="figure.tools.ruta.workbench.cde"> + <title>The UIMA Ruta cde framework + </title> + <mediaobject> + <imageobject role="html"> + <imagedata width="776px" format="PNG" align="center" + fileref="&imgroot;cde/cde.png" /> + </imageobject> + <imageobject role="fo"> + <imagedata width="5.4in" format="PNG" align="center" + fileref="&imgroot;cde/cde.png" /> + </imageobject> + <textobject> + <phrase>UIMA Ruta CDE perspective</phrase> + </textobject> + </mediaobject> + </figure> + <section id="section.tools.ruta.workbench.cde.documents"> + <title>CDE Documents view</title> + <para> + This view provides the general functionality to specify, which set of documents should be evaluated. The green start button in the toolbar + starts the evaluation process with the given configuration specified in the different views. + The text field <quote>Documents</quote> specifies the set of documents that have been annotated by an arbitrary model or set of rules. + These xmiCAS documents can contain additional annotations, which are used by the simple rule-based constraints. The text field <quote>Test Data</quote> + specifies an optional folder with gold standard documents. These documents can be applied for comparing the resulting CDE value to an F1 score. + The text field <quote>Type System</quote> refers to a type system descriptor, which defines all types that are needed for the evaluation. + Each text field is sensitive to drag and drop. The use can select a folder in the script explorer and drag and drop it on the text field. + Directly below the text fields, the values of different evaluation measures are given comparing the resulting CDE value to the F1 score. + These values are of course only available, if annotated gold standard data is specified in the text field <quote>Test Data</quote>. + </para> + <para> + The table at the bottom of the view contains all documents that are inspected. + The intervals specifying the color of the icon of each documents dependent of the CDE result can be set in the preference page. + If the evaluation has finished, then the CDE value is displayed and additionally the F1 score, if available. The table is sortable. + A double-click on a documents opens it in the CAS Editor. + </para> + </section> + <section id="section.tools.ruta.workbench.cde.constraints"> + <title>CDE Constraints view</title> + <para>This view specifies the currently applied constraints and their weights. This list is used to calculate the overall CDE value + of one document: the average of the weighted sum. The buttons in the toolbar of the view export and import the constraint using a simple xml format. + The buttons in the right part of the view can modify the list of the constraints. Currently, three types of constraints are supported: + Simple UIMA Ruta rules, a list of simple UIMA Ruta rules and word distribution constraints. All constraints return a value between 0 and 1. + The constraints based on UIMA Ruta rules return the ratio how often the rule was applied to how often the rules tried to apply. The rule + <quote>Author{STARTSWITH(Reference)};</quote>, for example, returns 1, if all author annotations start with a reference annotation. The word distribution + constraints refer to a txt file in which each line has the format <quote>"Proceedings":Booktitle 0.95, Journal 0.05</quote> specifying + the distribution of the word proceedings concerning the interesting annotations. If no quotes are given for the first term, then the term is interpreted as an annotation type. + The result value is currently calculated by using the cosine similarity of the expected and observed frequencies. + </para> + </section> + <section id="section.tools.ruta.workbench.cde.result"> + <title>CDE Result view</title> + <para> + The result view displays the result values of each constraint for the document that is selected in the CDE Documents view. + </para> + </section> +</section> Modified: uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/tools.ruta.workbench.xml URL: http://svn.apache.org/viewvc/uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/tools.ruta.workbench.xml?rev=1492177&r1=1492176&r2=1492177&view=diff ============================================================================== --- uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/tools.ruta.workbench.xml (original) +++ uima/sandbox/ruta/trunk/ruta-docbook/src/docbook/tools.ruta.workbench.xml Wed Jun 12 12:57:24 2013 @@ -45,6 +45,8 @@ under the License. <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="tools.ruta.workbench.explain_perspective.xml" /> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="tools.ruta.workbench.cde.xml" /> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="tools.ruta.workbench.query.xml" /> <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="tools.ruta.workbench.testing.xml" />