Hi Christophe, hi Barry, I've experimented a bit by including the sc-config switch, as Christophe mentioned. For me, this results in an aborted process even before the completion of the first iteration . Now the script fails to run extractor at ./mert-moses-multi.pl line 1374. I've also attached extract.err.BLEU and TER which might be helpful. Extract.err.TER says something like "Exception: std::bad_alloc". My call is
./mert-moses-multi.pl
/home/dan/smt/phrase/tuning/devset-1.tok.lw.en
/home/dan/smt/phrase/tuning/devset-1.tok.lw.de
/home/dan/smt/decoder/dist/cb5213a/bin/moses
/home/dan/smt/phrase/model/moses.ini
--working-dir /home/dan/smt/phrase/tuning/mert
--mertdir /home/dan/smt/decoder/mert
--rootdir /home/dan/smt/scripts/cb5213a
--decoder-flags="-v 0 -threads 2
--sc-config="BLEU:1,TER:3"
Using WORKING_DIR: /home/dan/smt/phrase/tuning/mert
Using SCRIPTS_ROOTDIR: /home/dan/smt/scripts/cb5213a
filtering the phrase tables... Tue Dec 20 15:37:02 CET 2011
exec: /home/dan/smt/scripts/cb5213a/training/filter-model-given-input.pl
./filtered /home/dan/smt/phrase/model/moses.ini
/home/dan/smt/phrase/tuning/m4loc/devset-1.tok.lw.en
Executing:
/home/dan/smt/scripts/cb5213a/training/filter-model-given-input.pl
./filtered /home/dan/smt/phrase/model/moses.ini
/home/dan/smt/phrase/tuning/devset-1.tok.lw.en > filterphrases.out 2>
filterphrases.err
Asking moses for feature names and values from filtered/moses.ini
Executing: /home/dan/smt/decoder/dist/cb5213a/bin/moses -v 0 -threads 2
-config filtered/moses.ini -inputtype 0 -show-weights > ./features.list
Creating lexical reordering...
weights: 0.200 0.200 0.200 0.200 0.200 0.200
Loading table into memory...done.
Creating lexical reordering...
weights: 0.200 0.200 0.200 0.200 0.200 0.200 0.200 0.200
Loading table into memory...done.
MERT starting values and ranges for random generation:
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
d = 0.200 ( 0.00 .. 1.00)
lm = 0.500 ( 0.00 .. 1.00)
w = -1.000 ( 0.00 .. 1.00)
tm = 0.200 ( 0.00 .. 1.00)
tm = 0.200 ( 0.00 .. 1.00)
tm = 0.200 ( 0.00 .. 1.00)
tm = 0.200 ( 0.00 .. 1.00)
tm = 0.200 ( 0.00 .. 1.00)
run 1 start at Tue Dec 20 15:38:23 CET 2011
Parsing --decoder-flags: |-v 0 -threads 2|
Saving new config to: ./run1.moses.ini
Saved: ./run1.moses.ini
(1) run decoder to produce n-best lists
params = -v 0 -threads 2
Normalizing lambdas: 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000
0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000
0.200000 0.500000 -1.000000 0.200000 0.200000 0.200000 0.200000 0.200000
DECODER_CFG = -w -0.181818 -lm 0.090909 -d 0.036364 0.036364 0.036364
0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364
0.036364 0.036364 0.036364 0.036364 -tm 0.036364 0.036364 0.036364 0.036364
0.036364
decoder_config = -w -0.181818 -lm 0.090909 -d 0.036364 0.036364 0.036364
0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364
0.036364 0.036364 0.036364 0.036364 -tm 0.036364 0.036364 0.036364 0.036364
0.036364
Executing: /home/dan/smt/decoder/dist/cb5213a/bin/moses -v 0 -threads 2
-config filtered/moses.ini -inputtype 0 -w -0.181818 -lm 0.090909 -d
0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364
0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 -tm 0.036364
0.036364 0.036364 0.036364 0.036364 -n-best-list run1.best100.out 100
-input-file /home/dan/smt/phrase/tuning/devset-1.tok.lw.en > run1.out
Creating lexical reordering...
weights: 0.036 0.036 0.036 0.036 0.036 0.036
Loading table into memory...done.
Creating lexical reordering...
weights: 0.036 0.036 0.036 0.036 0.036 0.036 0.036 0.036
Loading table into memory...done.
Translating line 0 in thread id 3041061744
reading bin ttable
size of OFF_T 8
Translating line 1 in thread id 3032669040
reading bin ttable
size of OFF_T 8
binary phrasefile loaded, default OFF_T: -1
binary phrasefile loaded, default OFF_T: -1
Translating line 2 in thread id 3032669040
...
Translating line 999 in thread id 3032669040
The decoder returns the scores in this order: d d d d d d d d d d d d d d d
lm w tm tm tm tm tm
Executing: gzip -f run1.best100.out
Scoring the nbestlist.
exec: /home/dan/smt/decoder/mert/extractor --scconfig case:true --scfile
run1.scores.dat.BLEU --ffile run1.features.dat.BLEU --sctype BLEU -r
/home/dan/smt/phrase/tuning/devset-1.tok.lw.de -n run1.best100.out.gz
Executing: /home/dan/smt/decoder/mert/extractor --scconfig case:true
--scfile run1.scores.dat.BLEU --ffile run1.features.dat.BLEU --sctype BLEU
-r /home/dan/smt/phrase/tuning/m4loc/devset-1.tok.lw.de -n
run1.best100.out.gz > extract.out.BLEU 2> extract.err.BLEU
exec: /home/dan/smt/decoder/mert/extractor --scconfig case:true --scfile
run1.scores.dat.TER --ffile run1.features.dat.TER --sctype TER -r
/home/dan/smt/phrase/tuning/m4loc/devset-1.tok.lw.de -n run1.best100.out.gz
Executing: /home/dan/smt/decoder/mert/extractor --scconfig case:true
--scfile run1.scores.dat.TER --ffile run1.features.dat.TER --sctype TER -r
/home/dan/smt/phrase/tuning/m4loc/devset-1.tok.lw.de -n run1.best100.out.gz
> extract.out.TER 2> extract.err.TER
Exit code: 1
ERROR: Failed to run '/home/dan/smt/decoder/mert/extractor --scconfig
case:true --scfile run1.scores.dat.TER --ffile run1.features.dat.TER
--sctype TER -r /home/dan/smt/phrase/tuning/m4loc/devset-1.tok.lw.de -n
run1.best100.out.gz'. at ./mert-moses-multi.pl line 1374.
Regards,
Daniel
-----Ursprüngliche Nachricht-----
Von: Daniel Schaut [mailto:[email protected]]
Gesendet: Dienstag, 20. Dezember 2011 15:05
An: 'Christophe Servan'; '[email protected]'
Cc: 'Barry Haddow'
Betreff: AW: [Moses-support] mert-moses-multi.pl: Failed to run mert at
./mert-moses-multi.pl line 1374.
Hi Barry, hi Christophe,
Thanks for your answers. Please find attached mert.out and mert.log.
>Why do you think it's running out of memory?
I assume my system ran out of memory, because when it failed to run mert,
memory usage was at 100% for quite a while. Don't know what happened
exactly. I'll try to perform some other runs and keep you updated.
Regards,
Daniel
-----Ursprüngliche Nachricht-----
Von: Christophe Servan [mailto:[email protected]]
Gesendet: Montag, 19. Dezember 2011 20:33
An: [email protected]; Daniel Schaut
Cc: Barry Haddow
Betreff: Re: [Moses-support] mert-moses-multi.pl: Failed to run mert at
./mert-moses-multi.pl line 1374.
Hi Daniel,
As Barry said, I made this variation of the mert-moses.pl in order to tune
with multiple metrics together.
The tuning is made with a linear ponderation of metrics, for example :
(1xBLEU+2xTER)/3
The setting is made with the switch --sc-config="BLEU:1,TER:2" (for my
previous example).
If you don't use this switch, you will tune only with BLEU (the default
metric for tuning).
As Barry proposed, would you like to post the mert.out and mert.log you
generated ?
Best regards,
Christophe
Le 19/12/2011 15:44, Barry Haddow a écrit :
> Hi Daniel
>
> Why do you think it's running out of memory? Could you post mert.out
> and mert.log ?
>
> Christophe Servan is the person who knows most about this script,
>
> cheers - Barry
>
> On Sunday 18 Dec 2011 18:49:46 Daniel Schaut wrote:
>> mert.out 2> mert.log
>>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
extract.err.BLEU
Description: Binary data
extract.err.TER
Description: Binary data
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
