Hi all.
I mean the decoder "moses_chart" and the script "mert-moses-new.pl" in
the mt3_chart branch. (I follow "Compiling Chart Decoder" in
http://www.statmt.org/moses/?n=Development.GetStarted)
Thanks to John, I try the option "--no-filter-phrase-table" and it works
in a small size corpus. It seems that there is no difference between
"mert-moses.pl" and "mert-moses-new.pl" with the option, i.e.
"mert-moses.pl" also supports the option.
There is an another problem during the tree-based decoding. If a source
sentence has a word "X", then "moses_chart" gives a segmentation fault
as following.
Translating: <s> X 1 の 好ましい 範囲 は 、 15 〜 30 ％ まで の 深 さ で ある 。 </s> |||
[0,0]=X (1) [0,1]=X (1) [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) ...ellipsis...
[0..0]=1 [1..1]=2 [2..2]=50 [3..3]=200 ...ellipsis... [0..1]=2
[1..2]=Segmentation fault
Exit code: 139
The decoder died. CONFIG WAS -w 0.000000 -lm 0.500000 -d 0.500000
It doesn't when I replace "X" with another word such as "-X-".
2009-12-29 (화), 21:00 +0000, Hieu Hoang:
> You mean the decoder in the mt3_chart branch?
>
> I'm not sure the
> mert-moses-new.pl
> works yet. I think only the
> mert-moses.pl
> is working.
>
> also, make sure you're using the script in the mt_chart branch,
> there's niggly small differences.
>
> 2009/12/29 John Morgan <[email protected]>
> There is an option --no-filter-phrase-table for
> mert-moses-new.pl.
>
>
>
> On 12/29/09, Hwidong Na <[email protected]> wrote:
> > I also have a problem with tuning a tree-based system.
> >
> > I use a parallel text without syntactic annotation to train
> the
> > tree-based system. Then the extracted rule-table has the
> only
> > non-terminal on both source and target side. (in
> Japanese-to-English)
> >
> > $ gzip -dc <path-to-model>/rule-table.gz | sort -k8
> -t'|' -nr | head
> > -n5
> > [X] [X] ||| 高 開口 数 による 高 解像度 の ||| with
> high resolution by means of
> > a high numerical aperture ||| ||| 1 2.07446e-05 1
> 2.2417e-08 2.718 |||
> > 0.0232558 0.0232558
> > [X] [X] ||| 高 開口 数 による 高 [X] ||| with high
> [X] high numerical
> > aperture ||| 5-2 ||| 1 0.000225055 0.5 0.000282326 2.718 |||
> 0.0232558
> > 0.0465116
> > [X] [X] ||| 高 開口 数 による 高 [X] ||| with high
> [X] a high numerical
> > aperture ||| 5-2 ||| 1 0.000225055 0.5 1.64576e-05 2.718 |||
> 0.0232558
> > 0.0465116
> > [X] [X] ||| 高 開口 数 [X] 解像度 の ||| [X]
> resolution by means of a high
> > numerical aperture ||| 3-0 ||| 1 0.00589926 1 5.15592e-07
> 2.718 |||
> > 0.0232558 0.0232558
> > [X] [X] ||| 高 開口 数 [X] の ||| [X] of a high
> numerical aperture |||
> > 3-0 ||| 1 0.0503172 1 0.000170465 2.718 ||| 0.0232558
> 0.0232558
> >
> >
> > Then passing "moses_chart" to "mert-moses-new" gives
> something wrong.
> > The first step of decoding is filtering of
> translation-table. The output
> > message of filtering scripts shows
> >
> > filtering <path-to-model>/rule-table.gz ->
> > <path-to-tuning>/filtered/phrase-table.0-0.1...
> > 0 of 550276 phrases pairs used (0.00%) - note: max
> length 10
> > filtering <path-to-model>/glue-grammar ->
> > <path-to-tuning>/filtered/phrase-table.0-0.2...
> > 0 of 3 phrases pairs used (0.00%) - note: max length
> 10
> >
> > It seems wrong. So how can I filter correctly? (or just do
> not filter?)
> >
> >> From: Philipp Koehn <pko...@...>
> >> Subject: Re: tuning tree-based models
> >> Newsgroups: gmane.comp.nlp.moses.user
> >> Date: 2009-12-28 23:56:11 GMT (12 hours and 2 minutes ago)
> >>
> >> Hi,
> >>
> >> this should work... Does the moses process generate a
> proper n-best list
> >> file?
> >> There may be something wrong with running the decoder.
> >>
> >> Regarding the section "non-terminals" in the moses.ini file
> - don't worry,
> >> this
> >> is just a list of special non-terminals that are used for
> unknown words
> >> etc.
> >>
> >> -phi
> >>
> >> On Mon, Dec 28, 2009 at 2:37 PM, John Morgan
> >> <johnjosephmor...@...> wrote:
> >> > Hello All,
> >> > I want to tune a tree-based trained system. I pass the
> moses_chart
> >> > decoder to the mert-moses-new.perl script. The script
> seems to make 1
> >> > decoding run producing features.dat and scores.dat files,
> but it
> >> > writes an empty extract.out file. Then the script stops.
> >> > I also noticed that the train-moses.perl script only
> writes out a "X"
> >> > under [non-terminals] in the moses.ini file. I have
> several
> >> > non-terminals on the input side.
> >> > Thanks,
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > John J Morgan
> >> > _______________________________________________
> >> > Moses-support mailing list
> >> > moses-supp...@...
> >> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >> >
> > --
> > Hwidong Na <[email protected]>
> > POSTECH
> >
> >
> >
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > [email protected]
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
>
>
>
> --
> Regards,
> John J Morgan
>
> _______________________________________________
> Moses-support mailing list
>
>
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
--
Hwidong Na <[email protected]>
POSTECH
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
