[
https://issues.apache.org/jira/browse/OPENNLP-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabriele Vaccari updated OPENNLP-1163:
--------------------------------------
Description:
The Sentence Detector trained with an abbreviations list (see attachment) fails
to spot them within a text if they are preceded by a punctuation mark.
In Italian, words starting with a vowel may be preceded by an article plus
apostrophe sign (single quote). Example: L'ARTICOLO (the article). The term
ARTICOLO, especially in legal text, is frequently abbreviated to ART.
Repro steps:
1) add the "art." abbreviation in the abbreviations XML file (enclosed, ctrl+F
"art.", case insensitive)
2) train a model for the Italian language (training set enclosed) with the
following command:
opennlp SentenceDetectorTrainer -abbDict "it-abbr.txt" -lang it -model
it-sen.bin -data training-set.txt -encoding UTF-8
3) run the model against a test text with the following command:
opennlp SentenceDetector it-sen.bin < test.txt
Even though the abbreviation "art." was included in the XML file, the sentence
detector breaks the sentence on instances of this abbreviation preceded by
article and apostrophe (e.g. nell'art., dall'art., dell'art.). See also the
enclosed output file out.txt, lines 6-7, 12-13, 13-14 and 16-17.
was:
The Sentence Detector trained with an abbreviations list (see attachment) fails
to spot them within a text if they are preceded by a punctuation mark.
In Italian, words starting with a vowel may be preceded by an article plus
apostrophe sign (single quote). Example: L'ARTICOLO (the article). The term
ARTICOLO, especially in legal text, is frequently abbreviated to ART.
Repro steps:
1) add the ART. abbreviation in the abbreviations XML file (enclosed, ctrl+F
"art.")
2) train a model for the Italian language (training set enclosed) with the
following command:
opennlp SentenceDetectorTrainer -abbDict "it-abbr.txt" -lang it -model
it-sen.bin -data training-set.txt -encoding UTF-8
3) run the model against a test text with the following command:
opennlp SentenceDetector it-sen.bin < test.txt
Even though the abbreviation "art." was included in the XML file, the sentence
detector breaks the sentence on instances of this abbreviation preceded by
article and apostrophe (e.g. nell'art., dall'art., dell'art.). See also the
enclosed output file out.txt, lines 6-7, 12-13, 13-14 and 16-17.
> Sentence detector doesn't spot abbreviations next to punctuation
> ----------------------------------------------------------------
>
> Key: OPENNLP-1163
> URL: https://issues.apache.org/jira/browse/OPENNLP-1163
> Project: OpenNLP
> Issue Type: Bug
> Components: Sentence Detector
> Affects Versions: 1.8.3
> Environment: Reproduced on Windows 10
> Reporter: Gabriele Vaccari
> Priority: Critical
> Labels: abbreviation, sentence-detector
> Attachments: it-abbr.txt, out.txt, test.txt, training-set.txt
>
>
> The Sentence Detector trained with an abbreviations list (see attachment)
> fails to spot them within a text if they are preceded by a punctuation mark.
> In Italian, words starting with a vowel may be preceded by an article plus
> apostrophe sign (single quote). Example: L'ARTICOLO (the article). The term
> ARTICOLO, especially in legal text, is frequently abbreviated to ART.
> Repro steps:
> 1) add the "art." abbreviation in the abbreviations XML file (enclosed,
> ctrl+F "art.", case insensitive)
> 2) train a model for the Italian language (training set enclosed) with the
> following command:
> opennlp SentenceDetectorTrainer -abbDict "it-abbr.txt" -lang it -model
> it-sen.bin -data training-set.txt -encoding UTF-8
> 3) run the model against a test text with the following command:
> opennlp SentenceDetector it-sen.bin < test.txt
> Even though the abbreviation "art." was included in the XML file, the
> sentence detector breaks the sentence on instances of this abbreviation
> preceded by article and apostrophe (e.g. nell'art., dall'art., dell'art.).
> See also the enclosed output file out.txt, lines 6-7, 12-13, 13-14 and 16-17.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
