Hi Hwidong
Can you send me a few lines of the phrase table for the hierarchical model.
The 1 change that we've done in the last few weeks which is breaking
previous experiments is the phrase table format has changed from
source ||| target ||| [alignment] ||| scores ||| [count]
to
source ||| target ||| scores ||| [alignment] ||| [count]
eg.
[X][DT] [X][NN] [X] ||| [X][DT] [X][NN] [NP] ||| 0-0 1-1 ||| 1.0
||| 5442.66 323.246
to
[X][DT] [X][NN] [X] ||| [X][DT] [X][NN] [NP] ||| 1.0 ||| 0-0 1-1
||| 5442.66 323.246
to make the code easier & more consistent for the phrase-based and
hierarchical models. If you're using phrase tables created by the old
code, it may not work with the new release. But hopefully, it's easy to
change. Sorry about that
On 11/08/2010 04:34, Hwidong Na wrote:
> Hi all,
>
> I run the new version with the EMS. I get an error when I try a
> hierarchical model. The configuration for the hierarchical model works
> in the previous revision (r3284).
>
> $SCRIPTS_ROOTDIR/ems/experiment.perl -config config.hierarchical
>
> ...ellipsis...
> executing /home00/leona/IWSLT10.zh-en/steps/3/TUNING_tune.3 via
> sh (1)
> number of steps doable or running: 1
> step TUNING:tune crashed
> number of steps doable or running: 0
>
> The running script of TUNING:tune is
>
>
> /home/leona/moses-2010-08-09/bin/scripts-20100810-1203//training/mert-moses-new.pl
> -mertdir=/home/leona/moses-2010-08-09/mert
> /home00/leona/IWSLT10.zh-en/tuning/input.txt.1
> /home00/leona/IWSLT10.zh-en/tuning/reference.tok.1.ref
> /home/leona/moses-2010-08-09/moses-chart-cmd/src/moses_chart
> /home00/leona/IWSLT10.zh-en/model/moses.ini.2 --nbest 100 --working-dir
> /home00/leona/IWSLT10.zh-en/tuning/tmp.3 --decoder-flags " -v 0 " --rootdir
> /home/leona/moses-2010-08-09/bin/scripts-20100810-1203/
> --no-filter-phrase-table
>
> In the STDERR of TUNING:tune,
>
> After default: -l mem_free=0.5G -hard
> Using
> SCRIPTS_ROOTDIR:
> /home/leona/moses-2010-08-09/bin/scripts-20100810-1203/
> Executing: mkdir -p /home00/leona/IWSLT10.zh-en/tuning/tmp.3
> Saved: ./run1.moses.ini
> Normalizing lambdas: 0 1 1 0.3 0.2 0.3 0.2 0 0.3
> DECODER_CFG = -w %.6f -lm %.6f -d %.6f -tm %.6f %.6f %.6f %.6f
> %.6f %.6f
> values = 0 0.303030303030303 0.303030303030303
> 0.0909090909090909 0.0606060606060606 0.0909090909090909
> 0.0606060606060606 0 0.0909090909090909
> Executing:
> /home/leona/moses-2010-08-09/moses-chart-cmd/src/moses_chart -v 0 -config
> /home00/leona/IWSLT10.zh-en/model/moses.ini.2 -inputtype 0 -w 0.000000 -lm
> 0.303030 -d 0.303030 -tm 0.090909 0.060606 0.090909 0.060606 0.000000
> 0.090909 -n-best-list run1.best100.out 100 -i
> /home00/leona/IWSLT10.zh-en/tuning/input.txt.1> run1.out
> /home00/leona/IWSLT10.zh-en/lm/iwslt10.lm.1: line 23: warning:
> non-zero probability for<unk> in closed-vocabulary LM
> Start loading new format pt model : [0.000] seconds
> Start loading new format pt model : [70.000] seconds
> [0..0]= [1..1]= [2..2]= [3..3]= [4..4]= [5..5]= [6..6]= [7..7]=
> [8..8]= [9..9]= [10..10]= [11..11]= [12..12]= [13..13]=
> [14..14]= [15..15]= [16..16]= [17..17]= [0..1]= [1..2]= [2..3]=
> [3..4]= [4..5]= [5..6]= [6..7]= [7..8]= [8..9]= [9..10]=
> [10..11]=
> ...ellipsis...
> [10..11]= [11..12]= [12..13]= [13..14]= [0..2]= [1..3]= [2..4]=
> [3..5]= [4..6]= [5..7]= [6..8]= [7..9]= [8..10]= [9..11]=
> [10..12]= [11..13]= [12..14]= [0..3]= [1..4]= [2..5]= [3..6]=
> [4..7]=Segmentation fault
> Exit code: 139
>
> The decoder died. CONFIG WAS -w 0.000000 -lm 0.303030 -d
> 0.303030 -tm 0.090909 0.060606 0.090909 0.060606 0.000000
> 0.090909
>
> By the way, the new version works with EMS when I try a phrase-based
> model with a lexicalized reordering model. Here are differences in
> configurations.
>
> $diff config.hierarchical config.moseslex
> 36c36
> < decoder = $moses-src-dir/moses-chart-cmd/src/moses_chart
> ---
> > decoder = $moses-src-dir/moses-cmd/src/moses
> 266c266
> < #lexicalized-reordering = msd-bidirectional-fe
> ---
> > lexicalized-reordering = msd-bidirectional-fe
> 270c270
> < hierarchical-rule-set = true
> ---
> > #hierarchical-rule-set = true
>
> What is the problem?
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
