Re: Problems training my own sentence splitter, with dictionary

[Riccardo Tasso]

Hi William,
    the upgrade solved my problem with the serialization of the model, 
thank you.

Another question about dictionaries: I'm interested to implement my own 
Dictionary classes (especially for POSDictionary) whith its own backend 
(e.g. Redis instead of memory). It wouldn't be better if train method 
had a Dictionary interface, instead of a class as a parameter?

Thank you,
    Riccardo

On 27/09/2011 16:17, william.co...@gmail.com wrote:
Hi, Riccardo,

We improved a little how the abbreviation dictionary is handled in OpenNLP
1.5.2. You should try it using the current release candidate:
http://people.apache.org/~**joern/releases/opennlp-1.5.2-**incubating/rc1/<http://people.apache.org/~joern/releases/opennlp-1.5.2-incubating/rc1/>

1.5.2 includes a command line tool named DictionaryBuilder:

$ bin/opennlp DictionaryBuilder
Usage: opennlp DictionaryBuilder -inputFile in -outputFile out [-encoding
charsetName]

Arguments description:
-inputFile in
  Plain file with one entry per line
-outputFile out
  The dictionary file.
-encoding charsetName
  specifies the encoding which should be used for reading and writing text.
If not specified the system default will be used.

This tool should be used to create an abbreviation dictionary in XML format
as expected by the SentenceDetector and Tokenizer tools.

Regards,
William


On Tue, Sep 27, 2011 at 10:37 AM, Riccardo Tasso
<riccardo.ta...@gmail.com>wrote:

I'm trying to use OpenNLP to train a sentence splitter model, following the
manual, and everything is ok with it.

I've noticed that the train method supports a Dictionary of abbreviations,
which tipically give me problems with my sentence splitter and I wanted to
try this strategy.

1) If I train a sentence splitter model passing a Dictionary I've build and
I serialize it (just as I serialize a simpler model) I have problems in
loading the model from file:

SentenceModel model = SentenceDetectorME.train(**language, sampleStream,
true, abbreviations, 5, 100);
modelOut = new BufferedOutputStream(new FileOutputStream("/home/lib/**
apache-opennlp-1.5.1-**incubating/models/it/it-sent.**bin"));
model.serialize(modelOut);
[...]
modelIn = new FileInputStream("/home/lib/**apache-opennlp-1.5.1-**
incubating/models/it/it-sent.**bin");
final SentenceModel sentenceModel = new SentenceModel(modelIn);

i. e. the last instruction gives me the following exception:
java.io.IOException: Stream closed
    at java.util.zip.ZipInputStream.**ensureOpen(ZipInputStream.**java:61)
    at java.util.zip.ZipInputStream.**closeEntry(ZipInputStream.**java:108)
    at opennlp.tools.util.model.**BaseModel.<init>(BaseModel.**java:137)
    at opennlp.tools.sentdetect.**SentenceModel.<init>(**
SentenceModel.java:77)
    at SentenceDetector.main(**SentenceDetector.java:18)

2) I tried to skip the serialization/de-serialization phase, to go on with
my tests:
SentenceModel model = SentenceDetectorME.train(**language, sampleStream,
true, abbreviations, 5, 100);
String[] sentences = sentenceDetector.sentDetect(**document);

However sentences are splitted also on abbreviations which I declared in my
Dictionary, which isn't exactly what I expected. E.g. : "Sono Mr. Brown" is
uncorrectly splitted in "Sono Mr." and "Brown.".

Can you help me with these two problems, which seems to be different, but
may share the same issue.

Thanks,
    Riccardo