###These Baseline commands for IRSTLM work fine using ver 1.0 Moses. A good LM file is generated: ----------Baseline script is ok cd ../../apps/moses/mosesInstalls export IRSTLM=/apps/moses/mosesInstalls/irstlm #Generate the LM file /apps/moses/mosesInstalls/irstlm/bin/add-start-end.sh < /apps/moses/fr-en/parallel-in/training/Europarl3.fr-en.clean.en > /apps/moses/fr-en/parallel-in/training/Europarl3.fr-en.sb.en export IRSTLM=$HOME/mosesInstalls/irstlm; /apps/moses/mosesInstalls/irstlm/bin/build-lm.sh -i /apps/moses/fr-en/parallel-in/training/Europarl3.fr-en.sb.en -n 5 -k 10 -t ./moses/tmp -p -s improved-kneser-ney -o /apps/moses/fr-en/parallel-in/training/Europarl3.fr-en.lm.en # CREATE THE ARPA FILE /apps/moses/mosesInstalls/irstlm/bin/compile-lm --text yes /apps/moses/fr-en/parallel-in/training/Europarl3.fr-en.lm.en.gz /apps/moses/fr-en/parallel-in/training/Europarl3.fr-en.arpa.en # BINARIZE THE ARPA FILE /apps/moses/mosesInstalls/mosesdecoder/bin/build_binary -i -p 1.5 probing /apps/moses/fr-en/parallel-in/training/Europarl3.fr-en.arpa.en /apps/moses/fr-en/parallel-in/training/Europarl3.fr-en.blm.en
-----------EMS script is NOT ok ###Trying to do same with EMS an LM as below, but it generates a temporary irstlm-build-tmp directory with stat4/ dict and ngram .gz files but no lm fie is generated after the tmp fie gets deleted. # irstlm lm-training = "$moses-script-dir/generic/trainlm-irst.perl -cores $cores -irst-dir $irstlm-dir -temp-dir $working-dir/lm" settings = "" # order of the language model order = 5 type = 1 ### Also tried adding below to EMS, but still no LM saved # irstlm lm-binarizer = $irstlm-dir/compile-lm # kenlm, also set type to 8 lm-binarizer = "$moses-bin-dir/build_binary -i" type = 8 ---------------
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
