Hi Hiue,
Unfortunately, I use the new scripts to create phrase tables.
The hierarchical model has the new format.
[X][X] 五 百 [X] ||| [X][X] five hundred [X] ||| 0.944295
0.44408 0.595991 0.478547 2.718 ||| 0-0 ||| 12.3925 19.6349
The phrase-based model also has the new format, although it does not
contain the alignment information.
五 百 日元 ||| five hundred yen ||| 0.933333 0.429554 0.7
0.372403 2.718 ||| ||| 15 20
I also checkout the latest svn trunk. Despite a little modification of
the configuration for the EMS (e.g. rename mert-moses-new.pl to
mert-moses.pl), it also fails to decode an input sentence using a
hierarchical model. Still, a phrase-based model has no problem.
I don't find the source of the problem yet.
--
Hwidong Na <[email protected]>
KLE lab, POSTECH, KOREA
2010-08-11 (수), 10:01 +0100, Hieu Hoang:
> Hi Hwidong
>
> Can you send me a few lines of the phrase table for the hierarchical model.
>
> The 1 change that we've done in the last few weeks which is breaking
> previous experiments is the phrase table format has changed from
> source ||| target ||| [alignment] ||| scores ||| [count]
> to
> source ||| target ||| scores ||| [alignment] ||| [count]
> eg.
> [X][DT] [X][NN] [X] ||| [X][DT] [X][NN] [NP] ||| 0-0 1-1 ||| 1.0
> ||| 5442.66 323.246
> to
> [X][DT] [X][NN] [X] ||| [X][DT] [X][NN] [NP] ||| 1.0 ||| 0-0 1-1
> ||| 5442.66 323.246
>
> to make the code easier & more consistent for the phrase-based and
> hierarchical models. If you're using phrase tables created by the old
> code, it may not work with the new release. But hopefully, it's easy to
> change. Sorry about that
>
>
> On 11/08/2010 04:34, Hwidong Na wrote:
> > Hi all,
> >
> > I run the new version with the EMS. I get an error when I try a
> > hierarchical model. The configuration for the hierarchical model works
> > in the previous revision (r3284).
> >
> > $SCRIPTS_ROOTDIR/ems/experiment.perl -config config.hierarchical
> >
> > ...ellipsis...
> > executing /home00/leona/IWSLT10.zh-en/steps/3/TUNING_tune.3 via
> > sh (1)
> > number of steps doable or running: 1
> > step TUNING:tune crashed
> > number of steps doable or running: 0
> >
> > The running script of TUNING:tune is
> >
> >
> > /home/leona/moses-2010-08-09/bin/scripts-20100810-1203//training/mert-moses-new.pl
> > -mertdir=/home/leona/moses-2010-08-09/mert
> > /home00/leona/IWSLT10.zh-en/tuning/input.txt.1
> > /home00/leona/IWSLT10.zh-en/tuning/reference.tok.1.ref
> > /home/leona/moses-2010-08-09/moses-chart-cmd/src/moses_chart
> > /home00/leona/IWSLT10.zh-en/model/moses.ini.2 --nbest 100 --working-dir
> > /home00/leona/IWSLT10.zh-en/tuning/tmp.3 --decoder-flags " -v 0 "
> > --rootdir /home/leona/moses-2010-08-09/bin/scripts-20100810-1203/
> > --no-filter-phrase-table
> >
> > In the STDERR of TUNING:tune,
> >
> > After default: -l mem_free=0.5G -hard
> > Using
> > SCRIPTS_ROOTDIR:
> > /home/leona/moses-2010-08-09/bin/scripts-20100810-1203/
> > Executing: mkdir -p /home00/leona/IWSLT10.zh-en/tuning/tmp.3
> > Saved: ./run1.moses.ini
> > Normalizing lambdas: 0 1 1 0.3 0.2 0.3 0.2 0 0.3
> > DECODER_CFG = -w %.6f -lm %.6f -d %.6f -tm %.6f %.6f %.6f %.6f
> > %.6f %.6f
> > values = 0 0.303030303030303 0.303030303030303
> > 0.0909090909090909 0.0606060606060606 0.0909090909090909
> > 0.0606060606060606 0 0.0909090909090909
> > Executing:
> > /home/leona/moses-2010-08-09/moses-chart-cmd/src/moses_chart -v 0
> > -config /home00/leona/IWSLT10.zh-en/model/moses.ini.2 -inputtype 0 -w
> > 0.000000 -lm 0.303030 -d 0.303030 -tm 0.090909 0.060606 0.090909 0.060606
> > 0.000000 0.090909 -n-best-list run1.best100.out 100 -i
> > /home00/leona/IWSLT10.zh-en/tuning/input.txt.1> run1.out
> > /home00/leona/IWSLT10.zh-en/lm/iwslt10.lm.1: line 23: warning:
> > non-zero probability for<unk> in closed-vocabulary LM
> > Start loading new format pt model : [0.000] seconds
> > Start loading new format pt model : [70.000] seconds
> > [0..0]= [1..1]= [2..2]= [3..3]= [4..4]= [5..5]= [6..6]= [7..7]=
> > [8..8]= [9..9]= [10..10]= [11..11]= [12..12]= [13..13]=
> > [14..14]= [15..15]= [16..16]= [17..17]= [0..1]= [1..2]= [2..3]=
> > [3..4]= [4..5]= [5..6]= [6..7]= [7..8]= [8..9]= [9..10]=
> > [10..11]=
> > ...ellipsis...
> > [10..11]= [11..12]= [12..13]= [13..14]= [0..2]= [1..3]= [2..4]=
> > [3..5]= [4..6]= [5..7]= [6..8]= [7..9]= [8..10]= [9..11]=
> > [10..12]= [11..13]= [12..14]= [0..3]= [1..4]= [2..5]= [3..6]=
> > [4..7]=Segmentation fault
> > Exit code: 139
> >
> > The decoder died. CONFIG WAS -w 0.000000 -lm 0.303030 -d
> > 0.303030 -tm 0.090909 0.060606 0.090909 0.060606 0.000000
> > 0.090909
> >
> > By the way, the new version works with EMS when I try a phrase-based
> > model with a lexicalized reordering model. Here are differences in
> > configurations.
> >
> > $diff config.hierarchical config.moseslex
> > 36c36
> > < decoder = $moses-src-dir/moses-chart-cmd/src/moses_chart
> > ---
> > > decoder = $moses-src-dir/moses-cmd/src/moses
> > 266c266
> > < #lexicalized-reordering = msd-bidirectional-fe
> > ---
> > > lexicalized-reordering = msd-bidirectional-fe
> > 270c270
> > < hierarchical-rule-set = true
> > ---
> > > #hierarchical-rule-set = true
> >
> > What is the problem?
>
>
>
>
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
