the X problem is strange. I can't reproduce the error, eg. for input X the man bought the house or with an X in the phrase table [X] [X] ||| X ||| hello ||| ||| 0.17
can you send me the ini file and a fuller trace of the output? thx hieu Hwidong Na wrote: > Hi all. > > I mean the decoder "moses_chart" and the script "mert-moses-new.pl" in > the mt3_chart branch. (I follow "Compiling Chart Decoder" in > http://www.statmt.org/moses/?n=Development.GetStarted) > > Thanks to John, I try the option "--no-filter-phrase-table" and it works > in a small size corpus. It seems that there is no difference between > "mert-moses.pl" and "mert-moses-new.pl" with the option, i.e. > "mert-moses.pl" also supports the option. > > There is an another problem during the tree-based decoding. If a source > sentence has a word "X", then "moses_chart" gives a segmentation fault > as following. > > Translating: <s> X 1 の 好ましい 範囲 は 、 15 〜 30 ％ まで の 深 さ で ある 。 </s> > ||| [0,0]=X (1) [0,1]=X (1) [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) ...ellipsis... > [0..0]=1 [1..1]=2 [2..2]=50 [3..3]=200 ...ellipsis... [0..1]=2 > [1..2]=Segmentation fault > Exit code: 139 > The decoder died. CONFIG WAS -w 0.000000 -lm 0.500000 -d 0.500000 > > It doesn't when I replace "X" with another word such as "-X-". > > 2009-12-29 (화), 21:00 +0000, Hieu Hoang: > >> You mean the decoder in the mt3_chart branch? >> >> I'm not sure the >> mert-moses-new.pl >> works yet. I think only the >> mert-moses.pl >> is working. >> >> also, make sure you're using the script in the mt_chart branch, >> there's niggly small differences. >> >> 2009/12/29 John Morgan <[email protected]> >> There is an option --no-filter-phrase-table for >> mert-moses-new.pl. >> >> >> >> On 12/29/09, Hwidong Na <[email protected]> wrote: >> > I also have a problem with tuning a tree-based system. >> > >> > I use a parallel text without syntactic annotation to train >> the >> > tree-based system. Then the extracted rule-table has the >> only >> > non-terminal on both source and target side. (in >> Japanese-to-English) >> > >> > $ gzip -dc <path-to-model>/rule-table.gz | sort -k8 >> -t'|' -nr | head >> > -n5 >> > [X] [X] ||| 高 開口 数 による 高 解像度 の ||| with >> high resolution by means of >> > a high numerical aperture ||| ||| 1 2.07446e-05 1 >> 2.2417e-08 2.718 ||| >> > 0.0232558 0.0232558 >> > [X] [X] ||| 高 開口 数 による 高 [X] ||| with high >> [X] high numerical >> > aperture ||| 5-2 ||| 1 0.000225055 0.5 0.000282326 2.718 ||| >> 0.0232558 >> > 0.0465116 >> > [X] [X] ||| 高 開口 数 による 高 [X] ||| with high >> [X] a high numerical >> > aperture ||| 5-2 ||| 1 0.000225055 0.5 1.64576e-05 2.718 ||| >> 0.0232558 >> > 0.0465116 >> > [X] [X] ||| 高 開口 数 [X] 解像度 の ||| [X] >> resolution by means of a high >> > numerical aperture ||| 3-0 ||| 1 0.00589926 1 5.15592e-07 >> 2.718 ||| >> > 0.0232558 0.0232558 >> > [X] [X] ||| 高 開口 数 [X] の ||| [X] of a high >> numerical aperture ||| >> > 3-0 ||| 1 0.0503172 1 0.000170465 2.718 ||| 0.0232558 >> 0.0232558 >> > >> > >> > Then passing "moses_chart" to "mert-moses-new" gives >> something wrong. >> > The first step of decoding is filtering of >> translation-table. The output >> > message of filtering scripts shows >> > >> > filtering <path-to-model>/rule-table.gz -> >> > <path-to-tuning>/filtered/phrase-table.0-0.1... >> > 0 of 550276 phrases pairs used (0.00%) - note: max >> length 10 >> > filtering <path-to-model>/glue-grammar -> >> > <path-to-tuning>/filtered/phrase-table.0-0.2... >> > 0 of 3 phrases pairs used (0.00%) - note: max length >> 10 >> > >> > It seems wrong. So how can I filter correctly? (or just do >> not filter?) >> > >> >> From: Philipp Koehn <pko...@...> >> >> Subject: Re: tuning tree-based models >> >> Newsgroups: gmane.comp.nlp.moses.user >> >> Date: 2009-12-28 23:56:11 GMT (12 hours and 2 minutes ago) >> >> >> >> Hi, >> >> >> >> this should work... Does the moses process generate a >> proper n-best list >> >> file? >> >> There may be something wrong with running the decoder. >> >> >> >> Regarding the section "non-terminals" in the moses.ini file >> - don't worry, >> >> this >> >> is just a list of special non-terminals that are used for >> unknown words >> >> etc. >> >> >> >> -phi >> >> >> >> On Mon, Dec 28, 2009 at 2:37 PM, John Morgan >> >> <johnjosephmor...@...> wrote: >> >> > Hello All, >> >> > I want to tune a tree-based trained system. I pass the >> moses_chart >> >> > decoder to the mert-moses-new.perl script. The script >> seems to make 1 >> >> > decoding run producing features.dat and scores.dat files, >> but it >> >> > writes an empty extract.out file. Then the script stops. >> >> > I also noticed that the train-moses.perl script only >> writes out a "X" >> >> > under [non-terminals] in the moses.ini file. I have >> several >> >> > non-terminals on the input side. >> >> > Thanks, >> >> > >> >> > >> >> > -- >> >> > Regards, >> >> > John J Morgan >> >> > _______________________________________________ >> >> > Moses-support mailing list >> >> > moses-supp...@... >> >> > http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > >> > -- >> > Hwidong Na <[email protected]> >> > POSTECH >> > >> > >> > >> > >> > >> > _______________________________________________ >> > Moses-support mailing list >> > [email protected] >> > http://mailman.mit.edu/mailman/listinfo/moses-support >> > >> >> >> >> -- >> Regards, >> John J Morgan >> >> _______________________________________________ >> Moses-support mailing list >> >> >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> >> _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
