Revision: 18463
http://sourceforge.net/p/gate/code/18463
Author: ian_roberts
Date: 2014-11-17 19:59:36 +0000 (Mon, 17 Nov 2014)
Log Message:
-----------
Documentation for majority vote consensus PRs.
Modified Paths:
--------------
gate/trunk/plugins/Crowd_Sourcing/src/gate/crowdsource/classification/MajorityVoteClassificationConsensus.java
gate/trunk/plugins/Crowd_Sourcing/src/gate/crowdsource/ne/MajorityVoteAnnotationConsensus.java
userguide/trunk/crowdsourcing.tex
Modified:
gate/trunk/plugins/Crowd_Sourcing/src/gate/crowdsource/classification/MajorityVoteClassificationConsensus.java
===================================================================
---
gate/trunk/plugins/Crowd_Sourcing/src/gate/crowdsource/classification/MajorityVoteClassificationConsensus.java
2014-11-17 19:09:51 UTC (rev 18462)
+++
gate/trunk/plugins/Crowd_Sourcing/src/gate/crowdsource/classification/MajorityVoteClassificationConsensus.java
2014-11-17 19:59:36 UTC (rev 18463)
@@ -32,7 +32,7 @@
import java.util.Map;
@CreoleResource(name = "Majority-vote consensus builder (classification)",
comment = "Process results of a crowd annotation task to find "
- + "where annotators agree and disagree.", helpURL =
"http://gate.ac.uk/userguide/sec:crowd:classification";)
+ + "where annotators agree and disagree.", helpURL =
"http://gate.ac.uk/userguide/sec:crowd:classification:adjudication";)
public class MajorityVoteClassificationConsensus
extends
AbstractLanguageAnalyser {
Modified:
gate/trunk/plugins/Crowd_Sourcing/src/gate/crowdsource/ne/MajorityVoteAnnotationConsensus.java
===================================================================
---
gate/trunk/plugins/Crowd_Sourcing/src/gate/crowdsource/ne/MajorityVoteAnnotationConsensus.java
2014-11-17 19:09:51 UTC (rev 18462)
+++
gate/trunk/plugins/Crowd_Sourcing/src/gate/crowdsource/ne/MajorityVoteAnnotationConsensus.java
2014-11-17 19:59:36 UTC (rev 18463)
@@ -29,7 +29,7 @@
import gate.creole.metadata.RunTime;
@CreoleResource(name = "Majority-vote consensus builder (annotation)", comment
= "Process results of a crowd annotation task to find "
- + "where annotators agree and disagree.", helpURL =
"http://gate.ac.uk/userguide/sec:crowd:annotation";)
+ + "where annotators agree and disagree.", helpURL =
"http://gate.ac.uk/userguide/sec:crowd:annotation:adjudication";)
public class MajorityVoteAnnotationConsensus extends AbstractLanguageAnalyser {
private static final long serialVersionUID = -983107328639648414L;
Modified: userguide/trunk/crowdsourcing.tex
===================================================================
--- userguide/trunk/crowdsourcing.tex 2014-11-17 19:09:51 UTC (rev 18462)
+++ userguide/trunk/crowdsourcing.tex 2014-11-17 19:59:36 UTC (rev 18463)
@@ -78,8 +78,8 @@
To access the GATE crowdsourcing tools, you must first load the
\verb!Crowd_Sourcing! plugin. This plugin provides four PR types, a ``job
-builder'' and ``results importer'' for each of the two supported styles of
-crowdsourcing job.
+builder'', ``results importer'' and ``consensus builder'' for each of the two
+supported styles of crowdsourcing job.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\sect[sec:crowd:classification]{Entity classification}
@@ -293,6 +293,78 @@
annotations will be created for new judgments only, you will not get duplicate
annotations for judgments that have already been processed.
+
+\subsect[sec:crowd:classification:adjudication]{Automatic adjudication}
+
+Once you have imported the human judgments from CrowdFlower back into GATE
+annotations, a common next step is to take the multiply-annotated entities and
+attempt to build a single ``consensus'' set of the classifications where enough
+of the annotators agree. The simplest form of automatic adjudication is the
+majority-vote method, where classifications are automatically accepted if at
+least $n$ out of $m$ annotators agree on them. Entities that do not have the
+required level of agreement cannot be accepted automatically and must be double
+checked somehow, either directly within GATE Developer or via a second round of
+crowdsourcing.
+
+This approach is implemented by the ``Majority-vote consensus builder
+(classification)'' PR. It takes the following runtime parameters:
+\begin{description}
+\item[originalEntityASName/entityAnnotationType] the annotation set and type
+ representing the original entities that were used to generate the units sent
+ for annotation by the crowd.
+\item[resultASName/resultAnnotationType] the annotation set and type where
+ annotations corresponding to the judgments of your annotators were created by
+ the results importer.
+\item[answerFeatureName] (default ``answer'') the name of the feature on each
+ result annotation that represents the answer selected by the annotator.
+\item[minimumAgreement] the minimum number of annotators who must agree on the
+ same option in order for it to be eligible for the consensus set. Usually
+ this threshold would be set at more than half the total number of judgments
+ for each entity, so at most one option can meet the threshold, but this is
+ not required. In any case, an entity is only ever eligible for the consensus
+ set if \emph{exactly one} option meets the threshold.
+\item[consensusASName] (default ``crowdConsensus'') the annotation set where
+ consensus annotations should be created, for the entities that meet the
+ agreement threshold.
+\item[disputeASName] (default ``crowdDispute'') the annotation set where
+ disputed annotations should be placed, for the entities that do not meet the
+ agreement threshold (if the threshold is less than half the total number of
+ annotators it is possible for more than one option to meet the threshold --
+ these cases will also be considered disputed).
+\item[noAgreementAction] what to do with disputed entities. This is discussed
+ in more detail below.
+\end{description}
+
+When run over a corpus, the PR will look at each of the original entity
+annotations in turn and count up the number of annotators who selected each of
+the available options (including common options specified at the job level).
+If there is \emph{exactly} one option on which at least \emph{minimumAgreement}
+annotators agree, then it will create a single annotation in the consensus
+annotation set, whose type is \emph{resultAnnotationType} and whose
+\emph{answerFeatureName} feature is the selected option.
+
+If there is no agreement (either no option meets the threshold, or more than
+one option in the case when the threshold is below 50\%), then the PRs action
+is determined by the \emph{noAgreementAction} parameter:
+\begin{description}
+\item[resolveLocally] all the judgment annotations are copied from the result
+ set into the dispute set, so they can be inspected locally in GATE Developer
+ (typically using the annotation stack view).
+\item[reAnnotateByCrowd] the \emph{original} entity annotation is copied to the
+ dispute set, with two modifications:
+ \begin{itemize}
+ \item its \verb!cf_unit! feature is removed
+ \item its \verb!options! feature is restricted to the subset of the original
+ options that were selected by at least one of the annotators in the first
+ round of annotation.
+ \end{itemize}
+\end{description}
+
+In the \emph{reAnnotateByCrowd} case, the resulting entity annotations are
+suitable to be imported into CrowdFlower again for another round of
+crowdsourced annotation, but this time highly ambiguous entities have a smaller
+set of choices that will be presented to the workers.
+
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\sect[sec:crowd:annotation]{Entity annotation}
@@ -473,15 +545,20 @@
first place (or at least, that there are the same \emph{number} of tokens
in the same order within each snippet as there were when the unit was
created).
+\item[annotateSpans] (boolean, default true) should adjacent tokens be merged
+ into a single spanning annotation?
\end{description}
When run, the results importer PR will call the CrowdFlower REST API to
retrieve the list of judgments for each unit in turn, and then create
annotations of the target type in the target annotation set (as configured by
the ``result'' runtime parameters) for each judgment, matching the tokens that
-the annotator selected. A run of adjacent tokens will be treated as a single
-annotation spanning from the start of the first to the end of the last token in
-the sequence. Each generated annotation will have the following features:
+the annotator selected. By default, a run of adjacent tokens will be treated
+as a single annotation spanning from the start of the first to the end of the
+last token in the sequence, but this can be disabled by setting
+\emph{annotateSpans} to \verb!false!, in which case each token will be
+annotated independently. Each generated annotation will have the following
+features:
\begin{description}
\item[cf\_judgment] the ``judgment ID'' -- the unique identifier assigned to
this judgment by CrowdFlower.
@@ -502,4 +579,44 @@
annotations will be created for new judgments only, you will not get duplicate
annotations for judgments that have already been processed.
+\subsect[sec:crowd:annotation:adjudication]{Automatic adjudication}
+
+Once you have imported the human judgments from CrowdFlower back into GATE
+annotations, a common next step is to take the multiply-annotated entities and
+attempt to build a single ``consensus'' set of the annotations where enough
+of the annotators agree. The simplest form of automatic adjudication is the
+majority-vote method, where annotations are automatically accepted if at
+least $n$ out of $m$ annotators agree on them. Entities that do not have the
+required level of agreement cannot be accepted automatically and must be double
+checked manually.
+
+This approach is implemented by the ``Majority-vote consensus builder
+(annotation)'' PR. It takes the following runtime parameters:
+\begin{description}
+\item[resultASName/resultAnnotationType] the annotation set and type where
+ annotations corresponding to the judgments of your annotators were created by
+ the results importer.
+\item[minimumAgreement] the minimum number of annotators who must agree on the
+ same annotation in order for it to be eligible for the consensus set.
Usually
+ this threshold would be set at more than half the total number of judgments
+ for each entity, but this is not required.
+\item[consensusASName] (default ``crowdConsensus'') the annotation set where
+ consensus annotations should be created, for the entities that meet the
+ agreement threshold.
+\item[disputeASName] (default ``crowdDispute'') the annotation set where
+ disputed annotations should be placed, for the entities that do not meet the
+ agreement threshold.
+\end{description}
+
+When run over a corpus, the PR inspects each group of co-extensive annotations
+of the target type in turn (the results importer PR will never create
+overlapping annotations from the same human judgment, so a group of result
+annotations with exactly the same span must represent judgments by different
+workers). If at least \emph{minimumAgreement} annotators agreed on the same
+annotation then a single new annotation of the \emph{resultAnnotationType}
+(with no features) is created in the consensus set. If the agreement threshold
+is not met, then \emph{all} the result annotations in this group are copied to
+the dispute set so they can be inspected in GATE Developer (typically using the
+annotation stack view).
+
% vim:ft=tex
This was sent by the SourceForge.net collaborative development platform, the
world's largest Open Source development site.
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
GATE-cvs mailing list
GATE-cvs@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/gate-cvs
