Akhilesh,
Your problems start all the way at the beginning with step (1.1) where
it says, "classes already in place, reusing". The function that
generates the "already in place, reusing" message tests if the file
exists and is non-zero length. Your first run created empty, but not
zero-length MGIZA++ .gz files in step 2.2. They're not zero-length
because of various metadata (e.g. time stamp) stored in the .gz header.
Therefore, the second run assumes they're good and skips to the next step.
You're re-running the same command line that outputs its results into a
previous run's folders. Steps 1, 2 and 4 reuse the previous run's
outputs. Step 5 fails to open files from the previous but the errors do
not terminate because the errors are inside a Perl fork. Step 6 also
encounters errors but again fails to terminate because the errors are
inside forks. Finally, train-model.perl terminates in step 7 because the
error is not in a fork.
The genesis of your terminal failure occurred the first time you ran
this command line, not the repeated attempt. My guess? you had a failure
in MGIZA++ in step 2.2. To find out for sure, you'll need delete the
output folder and start from the beginning. When it fails, search the
log for step 2.2 for "WARNING" and "ERROR" messages. Some of these
messages should be terminal but they are not. Instead, MGIZA++ creates
empty but not zero-length .gz alignment files. This fools downstream
step and you'll get another error in a later stage.
Good luck.
open the results of the previous steps by . It finds that the output
files of the previous steps are corrupted
On 4/17/2016 1:50 AM, [email protected] wrote:
Date: Sat, 16 Apr 2016 19:50:27 +0100
From: Akhilesh Gupta<[email protected]>
Subject: [Moses-support] TRAINING MERT ERROR
To:[email protected]
Hello Sir,
I'm trying to execute command to train the model as given in manual
provided onofficial website.
But I'm getting this error.
COMMAND:
hieu@hieu-VirtualBox:~/workspace/working$ nohup nice
/home/hieu/workspace/mosesdecoder/scripts/training/train-model.perl
-root-dir train -corpus /home/hieu/workspace/corpus/clean -f fr -e en
-alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
0:3:/home/hieu/workspace/lm/blm.en:8 -external-bin-dir
/home/hieu/workspace/working/bin/ -mgiza >& training.out &
OUTPUT:
nohup: ignoring input
Using SCRIPTS_ROOTDIR: /home/hieu/workspace/mosesdecoder/scripts
Using multi-thread GIZA
using gzip
(1) preparing corpus @ Sat Apr 16 19:42:06 BST 2016
Executing: mkdir -p /home/hieu/workspace/working/train/corpus
(1.0) selecting factors @ Sat Apr 16 19:42:06 BST 2016
(1.1) running mkcls @ Sat Apr 16 19:42:06 BST 2016
/home/hieu/workspace/working/bin/mkcls -c50 -n2
-p/home/hieu/workspace/corpus/clean.fr
-V/home/hieu/workspace/working/train/corpus/fr.vcb.classes opt
/home/hieu/workspace/working/train/corpus/fr.vcb.classes already in
place, reusing
(1.1) running mkcls @ Sat Apr 16 19:42:06 BST 2016
/home/hieu/workspace/working/bin/mkcls -c50 -n2
-p/home/hieu/workspace/corpus/clean.en
-V/home/hieu/workspace/working/train/corpus/en.vcb.classes opt
/home/hieu/workspace/working/train/corpus/en.vcb.classes already in
place, reusing
(1.2) creating vcb file /home/hieu/workspace/working/train/corpus/fr.vcb @
Sat Apr 16 19:42:06 BST 2016
(1.2) creating vcb file /home/hieu/workspace/working/train/corpus/en.vcb @
Sat Apr 16 19:42:06 BST 2016
(1.3) numberizing corpus
/home/hieu/workspace/working/train/corpus/fr-en-int-train.snt @ Sat Apr 16
19:42:06 BST 2016
/home/hieu/workspace/working/train/corpus/fr-en-int-train.snt already in
place, reusing
(1.3) numberizing corpus
/home/hieu/workspace/working/train/corpus/en-fr-int-train.snt @ Sat Apr 16
19:42:06 BST 2016
/home/hieu/workspace/working/train/corpus/en-fr-int-train.snt already in
place, reusing
(2) running giza @ Sat Apr 16 19:42:06 BST 2016
(2.1a) running snt2cooc fr-en @ Sat Apr 16 19:42:06 BST 2016
Executing: mkdir -p /home/hieu/workspace/working/train/giza.fr-en
Executing: /home/hieu/workspace/working/bin/snt2cooc
/home/hieu/workspace/working/train/giza.fr-en/fr-en.cooc
/home/hieu/workspace/working/train/corpus/en.vcb
/home/hieu/workspace/working/train/corpus/fr.vcb
/home/hieu/workspace/working/train/corpus/fr-en-int-train.snt
/home/hieu/workspace/working/bin/snt2cooc
/home/hieu/workspace/working/train/giza.fr-en/fr-en.cooc
/home/hieu/workspace/working/train/corpus/en.vcb
/home/hieu/workspace/working/train/corpus/fr.vcb
/home/hieu/workspace/working/train/corpus/fr-en-int-train.snt
END.
(2.1b) running giza fr-en @ Sat Apr 16 19:42:06 BST 2016
/home/hieu/workspace/working/bin/mgiza -CoocurrenceFile
/home/hieu/workspace/working/train/giza.fr-en/fr-en.cooc -c
/home/hieu/workspace/working/train/corpus/fr-en-int-train.snt -m1 5 -m2 0
-m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 -ncpus 4
-nodumps 1 -nsmooth 4 -o
/home/hieu/workspace/working/train/giza.fr-en/fr-en -onlyaldumps 1 -p0
0.999 -s /home/hieu/workspace/working/train/corpus/en.vcb -t
/home/hieu/workspace/working/train/corpus/fr.vcb
/home/hieu/workspace/working/train/giza.fr-en/fr-en.A3.final.gz seems
finished, reusing.
(2.1a) running snt2cooc en-fr @ Sat Apr 16 19:42:06 BST 2016
Executing: mkdir -p /home/hieu/workspace/working/train/giza.en-fr
Executing: /home/hieu/workspace/working/bin/snt2cooc
/home/hieu/workspace/working/train/giza.en-fr/en-fr.cooc
/home/hieu/workspace/working/train/corpus/fr.vcb
/home/hieu/workspace/working/train/corpus/en.vcb
/home/hieu/workspace/working/train/corpus/en-fr-int-train.snt
/home/hieu/workspace/working/bin/snt2cooc
/home/hieu/workspace/working/train/giza.en-fr/en-fr.cooc
/home/hieu/workspace/working/train/corpus/fr.vcb
/home/hieu/workspace/working/train/corpus/en.vcb
/home/hieu/workspace/working/train/corpus/en-fr-int-train.snt
END.
(2.1b) running giza en-fr @ Sat Apr 16 19:42:07 BST 2016
/home/hieu/workspace/working/bin/mgiza -CoocurrenceFile
/home/hieu/workspace/working/train/giza.en-fr/en-fr.cooc -c
/home/hieu/workspace/working/train/corpus/en-fr-int-train.snt -m1 5 -m2 0
-m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 -ncpus 4
-nodumps 1 -nsmooth 4 -o
/home/hieu/workspace/working/train/giza.en-fr/en-fr -onlyaldumps 1 -p0
0.999 -s /home/hieu/workspace/working/train/corpus/fr.vcb -t
/home/hieu/workspace/working/train/corpus/en.vcb
/home/hieu/workspace/working/train/giza.en-fr/en-fr.A3.final.gz seems
finished, reusing.
(3) generate word alignment @ Sat Apr 16 19:42:07 BST 2016
Combining forward and inverted alignment from files:
/home/hieu/workspace/working/train/giza.fr-en/fr-en.A3.final.{bz2,gz}
/home/hieu/workspace/working/train/giza.en-fr/en-fr.A3.final.{bz2,gz}
Executing: mkdir -p /home/hieu/workspace/working/train/model
Executing: /home/hieu/workspace/mosesdecoder/scripts/training/giza2bal.pl
-d "gzip -cd
/home/hieu/workspace/working/train/giza.en-fr/en-fr.A3.final.gz" -i "gzip
-cd /home/hieu/workspace/working/train/giza.fr-en/fr-en.A3.final.gz"
|/home/hieu/workspace/mosesdecoder/scripts/../bin/symal -alignment="grow"
-diagonal="yes" -final="yes" -both="yes" >
/home/hieu/workspace/working/train/model/aligned.grow-diag-final-and
symal: computing grow alignment: diagonal (1) final (1)both-uncovered (1)
skip=<0> counts=<993>
(4) generate lexical translation table 0-0 @ Sat Apr 16 19:42:07 BST 2016
(/home/hieu/workspace/corpus/clean.fr
,/home/hieu/workspace/corpus/clean.en,/home/hieu/workspace/working/train/model/lex)
reusing: /home/hieu/workspace/working/train/model/lex.f2e and
/home/hieu/workspace/working/train/model/lex.e2f
(5) extract phrases @ Sat Apr 16 19:42:07 BST 2016
/home/hieu/workspace/mosesdecoder/scripts/generic/extract-parallel.perl 2
split "sort "/home/hieu/workspace/mosesdecoder/scripts/../bin/extract
/home/hieu/workspace/corpus/clean.en /home/hieu/workspace/corpus/clean.fr
/home/hieu/workspace/working/train/model/aligned.grow-diag-final-and
/home/hieu/workspace/working/train/model/extract 7 orientation --model
wbe-msd --GZOutput
Executing:
/home/hieu/workspace/mosesdecoder/scripts/generic/extract-parallel.perl 2
split "sort "/home/hieu/workspace/mosesdecoder/scripts/../bin/extract
/home/hieu/workspace/corpus/clean.en /home/hieu/workspace/corpus/clean.fr
/home/hieu/workspace/working/train/model/aligned.grow-diag-final-and
/home/hieu/workspace/working/train/model/extract 7 orientation --model
wbe-msd --GZOutput
MAX 7 1 0
Started Sat Apr 16 19:42:07 2016
using gzip
isBSDSplit=0
Executing: mkdir -p /home/hieu/workspace/working/train/model/tmp.6041; ls
-l /home/hieu/workspace/working/train/model/tmp.6041
total=993 line-per-split=497
split -d -l 497 -a 7 /home/hieu/workspace/corpus/clean.en
/home/hieu/workspace/working/train/model/tmp.6041/target.split -d -l 497 -a
7 /home/hieu/workspace/working/train/model/aligned.grow-diag-final-and
/home/hieu/workspace/working/train/model/tmp.6041/align.split -d -l 497 -a
7 /home/hieu/workspace/corpus/clean.fr
/home/hieu/workspace/working/train/model/tmp.6041/source.merging extract /
extract.inv
gunzip -c
/home/hieu/workspace/working/train/model/tmp.6041/extract.0000000.gz
/home/hieu/workspace/working/train/model/tmp.6041/extract.0000001.gz |
LC_ALL=C sort -T /home/hieu/workspace/working/train/model/tmp.6041 2>>
/dev/stderr | gzip -c >
/home/hieu/workspace/working/train/model/extract.sorted.gz 2>> /dev/stderr
gunzip -c
/home/hieu/workspace/working/train/model/tmp.6041/extract.0000000.inv.gz
/home/hieu/workspace/working/train/model/tmp.6041/extract.0000001.inv.gz |
LC_ALL=C sort -T /home/hieu/workspace/working/train/model/tmp.6041 2>>
/dev/stderr | gzip -c >
/home/hieu/workspace/working/train/model/extract.inv.sorted.gz 2>>
/dev/stderr
gzip: /home/hieu/workspace/working/train/model/tmp.6041/extract.0000000.gz:
No such file or directory
gzip: /home/hieu/workspace/working/train/model/tmp.6041/extract.0000001.gz:
No such file or directory
gzip:
/home/hieu/workspace/working/train/model/tmp.6041/extract.0000000.inv.gz:
No such file or directory
gzip:
/home/hieu/workspace/working/train/model/tmp.6041/extract.0000001.inv.gz:
No such file or directory
Finished Sat Apr 16 19:42:07 2016
(6) score phrases @ Sat Apr 16 19:42:07 BST 2016
(6.1) creating table half
/home/hieu/workspace/working/train/model/phrase-table.half.f2e @ Sat Apr 16
19:42:07 BST 2016
/home/hieu/workspace/mosesdecoder/scripts/generic/score-parallel.perl 2
"sort "/home/hieu/workspace/mosesdecoder/scripts/../bin/score
/home/hieu/workspace/working/train/model/extract.sorted.gz
/home/hieu/workspace/working/train/model/lex.f2e
/home/hieu/workspace/working/train/model/phrase-table.half.f2e.gz 0
Executing:
/home/hieu/workspace/mosesdecoder/scripts/generic/score-parallel.perl 2
"sort "/home/hieu/workspace/mosesdecoder/scripts/../bin/score
/home/hieu/workspace/working/train/model/extract.sorted.gz
/home/hieu/workspace/working/train/model/lex.f2e
/home/hieu/workspace/working/train/model/phrase-table.half.f2e.gz 0
using gzip
Started Sat Apr 16 19:42:07 2016
/home/hieu/workspace/mosesdecoder/scripts/../bin/score
/home/hieu/workspace/working/train/model/tmp.6079/extract.0.gz
/home/hieu/workspace/working/train/model/lex.f2e
/home/hieu/workspace/working/train/model/tmp.6079/phrase-table.half.0000000.gz
2>> /dev/stderr
/home/hieu/workspace/working/train/model/tmp.6079/
run.0.sh/home/hieu/workspace/working/train/model/tmp.6079/run.1.shmv
/home/hieu/workspace/working/train/model/tmp.6079/phrase-table.half.0000000.gz
/home/hieu/workspace/working/train/model/phrase-table.half.f2e.gzmv: cannot
stat
'/home/hieu/workspace/working/train/model/tmp.6079/phrase-table.half.0000000.gz':
No such file or directory
Exit code: 1
ERROR: Scoring of phrases failed at
/home/hieu/workspace/mosesdecoder/scripts/training/train-model.perl line
1784.
(6.3) creating table half
/home/hieu/workspace/working/train/model/phrase-table.half.e2f @ Sat Apr 16
19:42:07 BST 2016
/home/hieu/workspace/mosesdecoder/scripts/generic/score-parallel.perl 2
"sort "/home/hieu/workspace/mosesdecoder/scripts/../bin/score
/home/hieu/workspace/working/train/model/extract.inv.sorted.gz
/home/hieu/workspace/working/train/model/lex.e2f
/home/hieu/workspace/working/train/model/phrase-table.half.e2f.gz --Inverse
1
Executing:
/home/hieu/workspace/mosesdecoder/scripts/generic/score-parallel.perl 2
"sort "/home/hieu/workspace/mosesdecoder/scripts/../bin/score
/home/hieu/workspace/working/train/model/extract.inv.sorted.gz
/home/hieu/workspace/working/train/model/lex.e2f
/home/hieu/workspace/working/train/model/phrase-table.half.e2f.gz --Inverse
1
using gzip
Started Sat Apr 16 19:42:07 2016
/home/hieu/workspace/mosesdecoder/scripts/../bin/score
/home/hieu/workspace/working/train/model/tmp.6097/extract.0.gz
/home/hieu/workspace/working/train/model/lex.e2f
/home/hieu/workspace/working/train/model/tmp.6097/phrase-table.half.0000000.gz
--Inverse 2>> /dev/stderr
/home/hieu/workspace/working/train/model/tmp.6097/
run.0.sh/home/hieu/workspace/working/train/model/tmp.6097/run.1.shgunzip -c
/home/hieu/workspace/working/train/model/tmp.6097/phrase-table.half.*.gz
2>> /dev/stderr| LC_ALL=C sort -T
/home/hieu/workspace/working/train/model/tmp.6097 | gzip -c >
/home/hieu/workspace/working/train/model/phrase-table.half.e2f.gz 2>>
/dev/stderr rm -rf /home/hieu/workspace/working/train/model/tmp.6097
Finished Sat Apr 16 19:42:07 2016
(6.6) consolidating the two halves @ Sat Apr 16 19:42:07 BST 2016
Executing:/home/hieu/workspace/mosesdecoder/scripts/../bin/consolidate
/home/hieu/workspace/working/train/model/phrase-table.half.f2e.gz
/home/hieu/workspace/working/train/model/phrase-table.half.e2f.gz
/dev/stdout | gzip -c >
/home/hieu/workspace/working/train/model/phrase-table.gz
/home/hieu/workspace/mosesdecoder/scripts/../bin/consolidate: error while
loading shared libraries: libboost_serialization.so.1.59.0: cannot open
shared object file: No such file or directory
Executing: rm -f
/home/hieu/workspace/working/train/model/phrase-table.half.*
(7) learn reordering model @ Sat Apr 16 19:42:07 BST 2016
(7.1) [no factors] learn reordering model @ Sat Apr 16 19:42:07 BST 2016
(7.2) building tables @ Sat Apr 16 19:42:07 BST 2016
Executing:
/home/hieu/workspace/mosesdecoder/scripts/../bin/lexical-reordering-score
/home/hieu/workspace/working/train/model/extract.o.sorted.gz 0.5
/home/hieu/workspace/working/train/model/reordering-table. --model "wbe msd
wbe-msd-bidirectional-fe"
Lexical Reordering Scorer
scores lexical reordering models of several types (hierarchical,
phrase-based and word-based-extraction
terminate called after throwing an instance of 'util::ErrnoException'
what(): util/file.cc:76 in int util::OpenReadOrThrow(const char*) threw
ErrnoException because `-1 == (ret = open(name, 00))'.
No such file or directory while opening
/home/hieu/workspace/working/train/model/extract.o.sorted.gz
Aborted (core dumped)
Exit code: 134
ERROR: Lexical reordering scoring failed at
/home/hieu/workspace/mosesdecoder/scripts/training/train-model.perl line
1922.
I guess the error starts with step5. Please help.
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
