Revision: 16264
http://gate.svn.sourceforge.net/gate/?rev=16264&view=rev
Author: adamfunk
Date: 2012-11-13 15:50:03 +0000 (Tue, 13 Nov 2012)
Log Message:
-----------
Updated user guide for OpenNLP PRs.
Modified Paths:
--------------
userguide/trunk/misc-creole.tex
userguide/trunk/recent-changes.tex
Modified: userguide/trunk/misc-creole.tex
===================================================================
--- userguide/trunk/misc-creole.tex 2012-11-13 15:10:47 UTC (rev 16263)
+++ userguide/trunk/misc-creole.tex 2012-11-13 15:50:03 UTC (rev 16264)
@@ -2515,35 +2515,34 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
OpenNLP provides java-based tools for sentence detection, tokenization,
-pos-tagging, chunking and parsing, named-entity detection, and
-coreference. See the OpenNLP website for details:
-\url{http://opennlp.sourceforge.net/}. The tools use the Maxent machine
-learning package. See \url{http://maxent.sourceforge.net/} for details.
+pos-tagging, chunking, parsing, named-entity detection, and coreference. See
+\htlink{https://opennlp.apache.org/}{the OpenNLP website} for details.
In order to use these tools as GATE processing resources, load the `OpenNLP'
-plugin via the Plugin Management Console. Alternatively, the OpenNLP system
-can be loaded from the GATE GUI by simply selecting the `Load OpenNLP System'
-menu item under the `File' menu. The OpenNLP PRs will be loaded, together with
-a pre-configured corpus pipeline application containing these PRs. This is
-similar to loading the ANNIE application with default values.
+plugin via the Plugin Management Console. Alternatively, the OpenNLP system for
+English can be loaded from the GATE GUI by simply selecting \emph{Applications}
+$\rightarrow$ \emph{Ready Made Applications} $\rightarrow$ \emph{OpenNLP}
+$\rightarrow$ \emph{OpenNLP IE System}. Two sample applications are also
+provided for Dutch and German in this plugin's \texttt{resources} directory,
+although you need to
\htlink{http://opennlp.sourceforge.net/models-1.5/}{download
+ the relevant models from Sourceforge}.
-We have integrated the following five processing resources:
-
+We have integrated five OpenNLP tools into GATE processing resources:
+%%
\begin{itemize}
-\item{OpenNlpTokenizer} Tokenizer PR
-\item{OpenNlpSentenceSplit} Sentence Splitter PR
-\item{OpenNlpPOS} POS Tagger PR
-\item{OpenNlpNameFinder} NER PR
-\item{OpenNlpChunker} Chunker PR
+\item OpenNLP Tokenizer
+\item OpenNLP Sentence Splitter
+\item OpenNLP POS Tagger
+\item OpenNLP Chunker
+\item OpenNLP NER (named entity recognition)
\end{itemize}
-Note that in general, these PRs can be mixed with other PRs of similar
-types. For example, you could create a pipeline that uses the OpenNLP
-Tokenizer, and the ANNIE POS Tagger. You may occasionally have
-problems with some combinations. Notes on compatibility and PR
-prerequisites are given for each PR in
-Section~\ref{sec:misc-creole:opennlp:opennlpprs}.
+In general, these PRs can be mixed with other PRs of similar types. For
example,
+you could create a pipeline that uses the OpenNLP Tokenizer, and the ANNIE POS
+Tagger. You may occasionally have problems with some combinations, and
different
+OpenNLP models use different POS tagsets. Notes on compatibility and PR
+prerequisites are given for each PR in the sections below.
Note also that some of the OpenNLP tools use quite large machine learning
models, which the PRs need to load into memory. You may find that you have
@@ -2555,149 +2554,128 @@
PRs. This is followed by a section which gives brief details of each
PR. For more details on each, see the OpenNLP
website,~\url{http://opennlp.sourceforge.net/}.
-
+%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\subsect[sec:misc-creole:opennlp:params]{Parameters common to all PRs}
+\subsect[sec:misc-creole:opennlp:loadtime]{Init parameters and models}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+Most OpenNLP PRs have a \textbf{model} parameter, a URL that points to a valid
+maxent model trained for the relevant tool. (The OpenNLP POS tagger no longer
+requires a separate dictionary file.)
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\subsubsect[sec:misc-creole:opennlp:loadtime]{Load-time Parameters}
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+The NER PR has a \textbf{config} parameter, a URL that points to a
configuration
+file, described in more detail in Section~\ref{sec:misc-creole:opennlp:ner};
the
+sample files \texttt{models/english/en-ner.conf} and
+\texttt{models/dutch/nl-ner.conf} can be easily copied, modified, and imitated.
-All OpenNLP PRs have a `model' parameter, which takes a URL. The URL
-should reference a valid Maxent model, or in the case of the Name Finder a
-directory containing a set of models for the different types of name sought.
-Default models can be found in the `models/english' directory. In addition, the
-OpenNlpPOS POS Tagger PR has a `dictionary' parameter, which also takes a URL,
-and a `dictionaryEncoding' parameter giving the character encoding of the
-dictionary file. The default can be found in the `models/english' directory.
-
For details of training new models (outside of the GATE framework),
see Section~\ref{sec:misc-creole:opennlp:models}
-
+%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\subsubsect[sec:misc-creole:opennlp:runtime]{Run-time Parameters}
+\subsect[sec:misc-creole:opennlp:tok]{OpenNLP Tokenizer}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-The OpenNLP PRs have runtime parameters to specify the annotation sets they
-should use for input and/or output. These are detailed below in the
-description of each PR, but all PRs will use the default unnamed annotation set
-unless told otherwise.
-
+%%
+This PR has no prerequisites. It adds \emph{Token} and \emph{SpaceToken}
+annotations to the \textbf{annotationSetName} run-time parameter's set. Both
+kinds of annotations get a feature \emph{source}=\emph{OpenNLP}, and
+\emph{Token} annotations get a \emph{string} feature with the underlying string
+as its value.
+%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\subsect[sec:misc-creole:opennlp:opennlpprs]{OpenNLP PRs}
+\subsect[sec:misc-creole:opennlp:sent]{OpenNLP Sentence Splitter}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
+%%
+This PR has no prerequisites. It adds Sentence annotations (with a feature and
+value \emph{source}=\emph{OpenNLP}) to the \textbf{annotationSetName} run-time
+parameter's set.
+%%%%%
+% If the OpenNLP sentence splitter returns no output (i.e. fails
+% to find any sentences), then GATE will add a single sentence annotation from
+% offset 0 to the end of the document. This is to prevent downstream PRs that
+% require sentence annotations from failing.
+%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\subsubsect[sec:misc-creole:opennlp:tok]{OpenNlpTokenizer - Tokenizer PR}
+\subsect[sec:misc-creole:opennlp:pos]{OpenNLP POS Tagger}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%
+This PR adds a \emph{category} feature to each \emph{Token} annotation.
-This PR adds Token annotations to the annotation set specified by the
-\texttt{annotationSetName} parameter.
+This PR requires \emph{Sentence} and \emph{Token} annotations to be present in
+the annotation set specified by \textbf{inputASName}. (They do not have to
come
+from OpenNLP PRs.) If the \textbf{outputASName} is different, this PR will
copy
+each \emph{Token} annotation and add the \emph{category} feature to the output
+copy.
-This PR does not require any other PR to be run beforehand. It creates
-annotations of type Token, with a feature and value `source=openNLP'
-and a string feature that takes the underlying string as its value.
-
+The tagsets vary according to the models.
+%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\subsubsect[sec:misc-creole:opennlp:sent]{OpenNlpSentenceSplit - Sentence
Splitter PR}
+\subsect[sec:misc-creole:opennlp:ner]{OpenNLP NER (NameFinder)}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%
+This PR finds standard named entities and adds annotations for them.
-This PR adds Sentence annotations to the annotation set specified by the
-\texttt{annotationSetName} parameter.
+This PR requires \emph{Sentence} and \emph{Token} annotations to be present in
+the annotation set specified by the \textbf{inputASName} run-time parameter.
+(They do not have to come from OpenNLP PRs.) The \emph{Token} annotations do
+not need to have a \emph{category} feature (so a POS tagger is not a
+prerequisite to this PR).
-This PR does not require any other PR to be run beforehand. It creates
-annotations of type Sentence, with a feature and value `source=openNLP'. If
-the OpenNLP sentence splitter returns no output (i.e. fails to find any
-sentences), then GATE will add a single sentence annotation from offset 0 to
-the end of the document. This is to prevent downstream PRs that require
-sentence annotations from failing.
-
+This PR creates annotations in the \textbf{outputASName} run-time parameter's
+set with types specified in the configuration file, whose URL was specified as
+an init parameter so it cannot be changed after initialization. (The file's
+contents, however, can be changed---reinitializing the PR clears out any models
+in memory, reloads the config file, and loads the models now specified in that
+file.) A configuration file should consist of two whitespace-separated
columns,
+as in this example.
+%%
+\begin{center}
+\begin{verbatim}
+en-ner-date.bin Date
+en-ner-location.bin Location
+en-ner-money.bin Money
+en-ner-organization.bin Organization
+en-ner-percentage.bin Percentage
+en-ner-person.bin Person
+en-ner-time.bin Time
+\end{verbatim}
+\end{center}
+%%
+The first entry in each row contains a path to a model file (relative to the
+directory where the config file is located, so in this example the models are
+all in the same directory with the config file), and the second contains the
+annotation type to be generated from that model. More than one model file can
+generate the same annotation type.
+%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\subsubsect[sec:misc-creole:opennlp:pos]{OpenNlpPOS - POS Tagger PR}
+\subsect[sec:misc-creole:opennlp:chunk]{OpenNLP Chunker}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%
+This PR marks noun, verb, and other chunks using features on \emph{Token}
+annotations.
-This PR adds a feature for Part Of Speech to Token annotations.
+This PR requires \emph{Sentence} and \emph{Token} annotations to be present in
+\textbf{inputASName} run-time parameter's set, and requires \emph{category}
+features on the \emph{Token} annotations (so a POS tagger is a prerequisite).
-This PR requires Sentence and Token annotations to be present in the annotation
-set specified by its \texttt{annotationSetName} parameter before it will work.
-These Sentence and Token annotations do not have to be from another OpenNLP PR.
-They could, for example, be from the ANNIE PRs. This PR adds a `category'
-feature to each Token, with the predicted Part Of Speech as value.
-
+If the \textbf{outputASName} and \textbf{inputASName} run-time parameters are
+the same, the PR adds a feature named according to the \textbf{chunkFeature}
+run-time parameter to each \emph{Token} annotation. If the annotation sets are
+different, the PR copies each \emph{Token} and adds the feature to the output
+copy. The feature uses the common BIO values, as in the following examples:
+%%
+\begin{description}
+\item\textbf{B-NP} token begins of a noun phrase;
+\item\textbf{I-NP} token is inside a noun phrase;
+\item\textbf{B-VP} token begins a verb phrase;
+\item\textbf{I-VP} token is inside a verb phrase;
+\item\textbf{O} token is outside any phrase;
+\item\textbf{B-PP} token begins a prepositional phrase;
+\item\textbf{B-ADVP} token begins an adverbial phrase.
+\end{description}
+%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\subsubsect[sec:misc-creole:opennlp:ner]{OpenNlpNameFinder - NER PR}
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-This PR finds standard Named Entities, adding them as annotations named for
-each named Entity type.
-
-This PR requires Sentence and Token annotations to be present in the annotation
-set specified by its \texttt{inputASName} parameter before it will work. These
-Sentence and Token annotations do not have to be from another OpenNLP PR. They
-could, for example, be from the ANNIE PRs. You may find, however, that not all
-pairings of Tokenizer and Sentence Splitter will work successfully.
-
-The Token annotations do not need to have a `category' feature. In
-other words, you do not need to run a POS Tagger before using this PR.
-
-This PR creates annotations for each named entity in the annotation set
-specified by the \texttt{outputASName} parameter, with a feature and value
-`source=openNLP'. It also adds a feature of `type' to each annotation, with
-the same values as the annotation type. Types include:
-
-\begin{itemize}
-\item person
-\item organization
-\item location
-\item date
-\item time
-\item percentage
-\end{itemize}
-
-For full details of all types, see the OpenNLP
-website,~\url{http://opennlp.sourceforge.net/}.
-
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\subsubsect[sec:misc-creole:opennlp:chunk]{OpenNlpChunker - Chunker PR}
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-This PR finds noun, verb, and other chunks, adding their position as
-features Token annotations.
-
-This PR requires Sentence and Token annotations to be present in the annotation
-set specified by the \texttt{annotationSetName} parameter before it will work.
-The Token annotations need to have a `category' feature. In other words, you
-also need to run a POS Tagger before using this PR. The Sentence and Token
-annotations (and `category' POS tag features) do not have to be from another
-OpenNLP PR. They could, for example, be from the ANNIE PRs.
-
-This PR creates features of type `chunk' for each token. The value of
-this feature define whether the token is at the beginning of a chunk,
-inside a chunk, or outside a chunk, using the standard BIO
-model. Example values and their interpretations are:
-
-\begin{itemize}
-\item \textbf{B-NP} Token is the first (beginning) token of a Noun Phrase
-\item \textbf{I-NP} Token is inside a Noun Phrase
-\item \textbf{B-VP} Token is the first (beginning) token of a Verb Phrase
-\item \textbf{I-VP} Token is inside a Verb Phrase
-\item \textbf{O} Token is outside any phrase
-\item \textbf{B-PP} Token is the first (beginning) token of a Prepositional
Phrase
-\item \textbf{B-ADVP} Token is the first (beginning) token of an Adverbial
Phrase
-\end{itemize}
-
-
-For full details of all chunk values, see the OpenNLP
-website,~\url{http://opennlp.sourceforge.net/}.
-
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsect[sec:misc-creole:opennlp:models]{Training new models}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-
+%%
Within the OpenNLP framework, new models can be trained for each of the
tools. By default, the GATE PRs use the standard Maxent models for English
which can be found in the plugin's `models/english' directory. The models are
@@ -2707,7 +2685,7 @@
use the file URL of your new model as a value for the `model' parameter of the
PR. For details on how to train models, see the Maxent website
\url{http://maxent.sourceforge.net/}.
-
+%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\sect[sec:misc-creole:boilerpipe]{Content Detection Using Boilerpipe}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Modified: userguide/trunk/recent-changes.tex
===================================================================
--- userguide/trunk/recent-changes.tex 2012-11-13 15:10:47 UTC (rev 16263)
+++ userguide/trunk/recent-changes.tex 2012-11-13 15:50:03 UTC (rev 16264)
@@ -31,6 +31,10 @@
markup, or XML dump files (such as those produced by Wikipedia).
See Section~\ref{sec:creole:mediawiki} for more details.
+The OpenNLP plugin has been completely revised to use Apache OpenNLP 1.5.2 and
+the corresponding set of models. See Section~\ref{sec:misc-creole:opennlp} for
+details.
+
\rcSubsect{October 2012}
The PRs defined in the ANNIE plugin are now described by annotations on the
Java classes
This was sent by the SourceForge.net collaborative development platform, the
world's largest Open Source development site.
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
GATE-cvs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/gate-cvs
