yeah, i did https://gist.github.com/anonymous/542275c2fdb817c02c6275eb227c467d#file-test-java-L38-L41 The constructor will be called one time.... The InSpanFeatureGenerator does a similar thing.
2016-10-31 14:11 GMT+01:00 William Colen <william.co...@gmail.com>: > OK, you need to find a way to avoid executing POS Tagger for each call. > Maybe you can store the output to a cache. > > 2016-10-31 10:49 GMT-02:00 Damiano Porta <damianopo...@gmail.com>: > > > hmm ok i load the postagger in the constructor of my custom > > FeatureGenerator, this is an example: > > > > https://gist.github.com/anonymous/542275c2fdb817c02c6275eb227c467d > > > > what do you think? > > > > 2016-10-31 13:16 GMT+01:00 William Colen <william.co...@gmail.com>: > > > > > Unfortunately I don't think the API supports this. You will need a > hack. > > > > > > 2016-10-30 12:59 GMT-02:00 Damiano Porta <damianopo...@gmail.com>: > > > > > > > Jorn > > > > what suffix should i use if i need a postagger model in a > > > FeatureGenerator? > > > > > > > > For dictionary i use mydictionary.dictionary as you told me. What > about > > > > postagger .bin? > > > > > > > > Thanks > > > > Damiano > > > > > > > > Il 29/Ott/2016 14:27, "Damiano Porta" <damianopo...@gmail.com> ha > > > scritto: > > > > > > > > > ok! thank you Jorn! > > > > > > > > > > 2016-10-29 13:54 GMT+02:00 Joern Kottmann <kottm...@gmail.com>: > > > > > > > > > >> The class has to be on your classpath otherwise it can't be > loaded. > > > > >> > > > > >> Jörn > > > > >> > > > > >> On Fri, 2016-10-28 at 22:59 +0200, Damiano Porta wrote: > > > > >> > Jorn, > > > > >> > as I wrote i have created the ner model, but when i try to use > it > > > via > > > > >> > "opennlp TokenNameFinder" tool it can't locate my custom > > > > >> > FeatureGenerator. > > > > >> > This is the output: > > > > >> > > > > > >> > bash opennlp TokenNameFinder /home/damiano/custom-ner-model.bin > > > > >> > Loading Token Name Finder model ... done (5,948s) > > > > >> > Exception in thread "main" > > > > >> > opennlp.tools.util.ext.ExtensionNotLoadedException: Unable to > > find > > > > >> > implementation for > > > > >> > opennlp.tools.util.featuregen.AdaptiveFeatureGenerator, > > > > >> > the class or service > > > > >> > com.damiano.parser.generator.SpanWindowFeatureGenerator could > not > > > be > > > > >> > located! > > > > >> > at > > > > >> > opennlp.tools.util.ext.ExtensionLoader. > > > instantiateExtension(Extension > > > > >> > Loader.java:122) > > > > >> > at > > > > >> > opennlp.tools.util.featuregen.GeneratorFactory$ > > > CustomFeatureGenerator > > > > >> > Factory.create(GeneratorFactory.java:582) > > > > >> > at > > > > >> > opennlp.tools.util.featuregen.GeneratorFactory. > > > createGenerator(Genera > > > > >> > torFactory.java:661) > > > > >> > at > > > > >> > opennlp.tools.util.featuregen.GeneratorFactory$ > > > AggregatedFeatureGener > > > > >> > atorFactory.create(GeneratorFactory.java:129) > > > > >> > at > > > > >> > opennlp.tools.util.featuregen.GeneratorFactory. > > > createGenerator(Genera > > > > >> > torFactory.java:661) > > > > >> > at > > > > >> > opennlp.tools.util.featuregen.GeneratorFactory$ > > > CachedFeatureGenerator > > > > >> > Factory.create(GeneratorFactory.java:171) > > > > >> > at > > > > >> > opennlp.tools.util.featuregen.GeneratorFactory. > > > createGenerator(Genera > > > > >> > torFactory.java:661) > > > > >> > at > > > > >> > opennlp.tools.util.featuregen.GeneratorFactory$ > > > AggregatedFeatureGener > > > > >> > atorFactory.create(GeneratorFactory.java:129) > > > > >> > at > > > > >> > opennlp.tools.util.featuregen.GeneratorFactory. > > > createGenerator(Genera > > > > >> > torFactory.java:661) > > > > >> > at > > > > >> > opennlp.tools.util.featuregen.GeneratorFactory.create( > > > GeneratorFactor > > > > >> > y.java:711) > > > > >> > at > > > > >> > opennlp.tools.namefind.TokenNameFinderFactory. > > > createFeatureGenerators > > > > >> > (TokenNameFinderFactory.java:153) > > > > >> > at > > > > >> > opennlp.tools.namefind.TokenNameFinderFactory. > > > createContextGenerator( > > > > >> > TokenNameFinderFactory.java:118) > > > > >> > at opennlp.tools.namefind.NameFinderME.<init>( > > NameFinderME.java:90) > > > > >> > at > > > > >> > opennlp.tools.cmdline.namefind.TokenNameFinderTool. > > > run(TokenNameFinde > > > > >> > rTool.java:59) > > > > >> > at opennlp.tools.cmdline.CLI.main(CLI.java:227) > > > > >> > > > > > >> > Why does the custom generator is not included on the model? > > > > >> > This is the my xml descriptor: > > > > >> > > > > > >> > <generators> > > > > >> > <cache> > > > > >> > <generators> > > > > >> > <window prevLength="6" nextLength="3"> > > > > >> > <tokenclass/> > > > > >> > </window> > > > > >> > <window prevLength="6" nextLength="3"> > > > > >> > <token/> > > > > >> > </window> > > > > >> > <definition/> > > > > >> > <prevmap/> > > > > >> > <bigram/> > > > > >> > <sentence begin="true" end="false"/> > > > > >> > <dictionary prefix="name" dict="names.dictionary"/> > > > > >> > <custom > > > > >> > class="com.damiano.parser.generator.SpanWindowFeatureGenerator" > > > > >> > leftTokens="6" rightTokens="3"/> > > > > >> > </generators> > > > > >> > </cache> > > > > >> > </generators> > > > > >> > > > > > >> > > > > > >> > Damiano > > > > >> > > > > > >> > 2016-10-28 14:00 GMT+02:00 Damiano Porta < > damianopo...@gmail.com > > >: > > > > >> > > > > > >> > > > > > > >> > > Pardon, my wrong, i forgot to change <dictionary prefix="city" > > > > >> > > dict="damiano"/> into <dictionary prefix="city" > > > > >> > > dict="damiano.dictionary"/> in my train.xml > > > > >> > > > > > > >> > > now it is working well! and the .bin has my dictionary too > > > > >> > > > > > > >> > > 2016-10-28 13:51 GMT+02:00 Damiano Porta < > > damianopo...@gmail.com > > > >: > > > > >> > > > > > > >> > > > > > > > >> > > > Jorn > > > > >> > > > i change the code as you told me, this exactly: > > > > >> > > > https://gist.github.com/anonymous/ > 8877b09d441d2e64c181fa9b5a > > > > >> > > > de4550#file-test-java-L15 > > > > >> > > > > > > > >> > > > but i get this error: > > > > >> > > > > > > > >> > > > opennlp.tools.namefind.TokenNameFinderModel$ > > > FeatureGeneratorCreat > > > > >> > > > ionError: > > > > >> > > > opennlp.tools.util.InvalidFormatException: No dictionary > > > resource > > > > >> > > > for > > > > >> > > > key: damiano > > > > >> > > > at opennlp.tools.namefind.TokenNameFinderFactory. > > createFeatureG > > > > >> > > > enerators(TokenNameFinderFactory.java:176) > > > > >> > > > at opennlp.tools.namefind.TokenNameFinderFactory. > > createContextG > > > > >> > > > enerator(TokenNameFinderFactory.java:118) > > > > >> > > > at > > > > >> > > > opennlp.tools.namefind.NameFinderME.train( > > > NameFinderME.java:333) > > > > >> > > > at com.damiano.parser.trainer.NER.compileNER(NER.java:185) > > > > >> > > > at com.damiano.parser.trainer.NER.main(NER.java:155) > > > > >> > > > > > > > >> > > > Caused by: opennlp.tools.util.InvalidFormatException: No > > > > >> > > > dictionary > > > > >> > > > resource for key: damiano > > > > >> > > > at opennlp.tools.util.featuregen. > > GeneratorFactory$DictionaryFea > > > > >> > > > tureGeneratorFactory.create(GeneratorFactory.java:251) > > > > >> > > > at opennlp.tools.util.featuregen. > > GeneratorFactory.createGenerat > > > > >> > > > or(GeneratorFactory.java:661) > > > > >> > > > at opennlp.tools.util.featuregen. > > GeneratorFactory$AggregatedFea > > > > >> > > > tureGeneratorFactory.create(GeneratorFactory.java:129) > > > > >> > > > at opennlp.tools.util.featuregen. > > GeneratorFactory.createGenerat > > > > >> > > > or(GeneratorFactory.java:661) > > > > >> > > > at opennlp.tools.util.featuregen. > > GeneratorFactory$CachedFeature > > > > >> > > > GeneratorFactory.create(GeneratorFactory.java:171) > > > > >> > > > at opennlp.tools.util.featuregen. > > GeneratorFactory.createGenerat > > > > >> > > > or(GeneratorFactory.java:661) > > > > >> > > > at opennlp.tools.util.featuregen. > > GeneratorFactory$AggregatedFea > > > > >> > > > tureGeneratorFactory.create(GeneratorFactory.java:129) > > > > >> > > > at opennlp.tools.util.featuregen. > > GeneratorFactory.createGenerat > > > > >> > > > or(GeneratorFactory.java:661) > > > > >> > > > at opennlp.tools.util.featuregen. > > GeneratorFactory.create(Genera > > > > >> > > > torFactory.java:711) > > > > >> > > > at opennlp.tools.namefind.TokenNameFinderFactory. > > createFeatureG > > > > >> > > > enerators(TokenNameFinderFactory.java:153) > > > > >> > > > ... 4 more > > > > >> > > > > > > > >> > > > 2016-10-28 12:55 GMT+02:00 Joern Kottmann < > kottm...@gmail.com > > >: > > > > >> > > > > > > > >> > > > > > > > > >> > > > > Try to rename the dictionary key to xyz.dictionary then > the > > > > >> > > > > serializer > > > > >> > > > > will > > > > >> > > > > be mapped correctly. > > > > >> > > > > > > > > >> > > > > Jörn > > > > >> > > > > > > > > >> > > > > On Thu, Oct 27, 2016 at 11:14 PM, Damiano Porta > > > <damianoporta@g > > > > >> > > > > mail.com> > > > > >> > > > > wrote: > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > Jorn i add the Dictionary here: > > > > >> > > > > > https://gist.github.com/anonymous/ > > > bc822fb0520c4c42b75748bf414 > > > > >> > > > > > 7da > > > > >> > > > > > 34#file-train-java-L15 > > > > >> > > > > > > > > > >> > > > > > And unfortunately i only see this error: > > > > >> > > > > > > > > > >> > > > > > java.lang.IllegalStateException: Missing serializer for > > > > >> > > > > > damiano > > > > >> > > > > > at > > > > >> > > > > > opennlp.tools.util.model.BaseModel.serialize(BaseModel. > > > java:6 > > > > >> > > > > > 10) > > > > >> > > > > > > > > > >> > > > > > I do not have other info. > > > > >> > > > > > Do i have to create a custom Serializer too? > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > 2016-10-27 22:04 GMT+02:00 Joern Kottmann < > > > kottm...@gmail.com > > > > >> > > > > > >: > > > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > On Thu, 2016-10-27 at 21:18 +0200, Joern Kottmann > wrote: > > > > >> > > > > > > > > > > > >> > > > > > > > On Tue, 2016-10-25 at 18:49 +0200, Damiano Porta > > wrote: > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > i am getting a strange error during the compiling > > of a > > > > >> > > > > > > > > NER model. > > > > >> > > > > > > > > Basically, the end of the build output is: > > > > >> > > > > > > > > > > > > >> > > > > > > > > 98: ... loglikelihood=-13340.018762351776 > > > > >> > > > > > > > > 0.999005934601099 > > > > >> > > > > > > > > 99: ... loglikelihood=-13258.358751926637 > > > > >> > > > > > > > > 0.9990120681028991 > > > > >> > > > > > > > > 100: ... loglikelihood=-13178.039964721707 > > > > >> > > > > > > > > 0.9990177634974279 > > > > >> > > > > > > > > Exception in thread "main" > > > > >> > > > > > > > > java.lang.IllegalStateException: > > > > >> > > > > Missing > > > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > serializer for *mydictionary* > > > > >> > > > > > > > > at > > > > >> > > > > > > > > opennlp.tools.util.model. > > > BaseModel.serialize(BaseModel. > > > > >> > > > > > > > > java: > > > > >> > > > > 610) > > > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > Can you please post the full exception stack trace? > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > And what is the name of they key you used for the > > > > >> > > > > > > dictionary? > > > > >> > > > > > > The dictionary serializers are only mapped by > extension. > > > > >> > > > > > > > > > > >> > > > > > > Jörn > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > >
