There is an option --no-filter-phrase-table for mert-moses-new.pl.
On 12/29/09, Hwidong Na <[email protected]> wrote: > I also have a problem with tuning a tree-based system. > > I use a parallel text without syntactic annotation to train the > tree-based system. Then the extracted rule-table has the only > non-terminal on both source and target side. (in Japanese-to-English) > > $ gzip -dc <path-to-model>/rule-table.gz | sort -k8 -t'|' -nr | head > -n5 > [X] [X] ||| 高 開口 数 による 高 解像度 の ||| with high resolution by means of > a high numerical aperture ||| ||| 1 2.07446e-05 1 2.2417e-08 2.718 ||| > 0.0232558 0.0232558 > [X] [X] ||| 高 開口 数 による 高 [X] ||| with high [X] high numerical > aperture ||| 5-2 ||| 1 0.000225055 0.5 0.000282326 2.718 ||| 0.0232558 > 0.0465116 > [X] [X] ||| 高 開口 数 による 高 [X] ||| with high [X] a high numerical > aperture ||| 5-2 ||| 1 0.000225055 0.5 1.64576e-05 2.718 ||| 0.0232558 > 0.0465116 > [X] [X] ||| 高 開口 数 [X] 解像度 の ||| [X] resolution by means of a high > numerical aperture ||| 3-0 ||| 1 0.00589926 1 5.15592e-07 2.718 ||| > 0.0232558 0.0232558 > [X] [X] ||| 高 開口 数 [X] の ||| [X] of a high numerical aperture ||| > 3-0 ||| 1 0.0503172 1 0.000170465 2.718 ||| 0.0232558 0.0232558 > > > Then passing "moses_chart" to "mert-moses-new" gives something wrong. > The first step of decoding is filtering of translation-table. The output > message of filtering scripts shows > > filtering <path-to-model>/rule-table.gz -> > <path-to-tuning>/filtered/phrase-table.0-0.1... > 0 of 550276 phrases pairs used (0.00%) - note: max length 10 > filtering <path-to-model>/glue-grammar -> > <path-to-tuning>/filtered/phrase-table.0-0.2... > 0 of 3 phrases pairs used (0.00%) - note: max length 10 > > It seems wrong. So how can I filter correctly? (or just do not filter?) > >> From: Philipp Koehn <pko...@...> >> Subject: Re: tuning tree-based models >> Newsgroups: gmane.comp.nlp.moses.user >> Date: 2009-12-28 23:56:11 GMT (12 hours and 2 minutes ago) >> >> Hi, >> >> this should work... Does the moses process generate a proper n-best list >> file? >> There may be something wrong with running the decoder. >> >> Regarding the section "non-terminals" in the moses.ini file - don't worry, >> this >> is just a list of special non-terminals that are used for unknown words >> etc. >> >> -phi >> >> On Mon, Dec 28, 2009 at 2:37 PM, John Morgan >> <johnjosephmor...@...> wrote: >> > Hello All, >> > I want to tune a tree-based trained system. I pass the moses_chart >> > decoder to the mert-moses-new.perl script. The script seems to make 1 >> > decoding run producing features.dat and scores.dat files, but it >> > writes an empty extract.out file. Then the script stops. >> > I also noticed that the train-moses.perl script only writes out a "X" >> > under [non-terminals] in the moses.ini file. I have several >> > non-terminals on the input side. >> > Thanks, >> > >> > >> > -- >> > Regards, >> > John J Morgan >> > _______________________________________________ >> > Moses-support mailing list >> > moses-supp...@... >> > http://mailman.mit.edu/mailman/listinfo/moses-support >> > > -- > Hwidong Na <[email protected]> > POSTECH > > > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > -- Regards, John J Morgan _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
