Ok, thanks Daniel for the logs, I'll check this problem today. Christophe
Le 20/12/2011 19:40, Daniel Schaut a écrit : > Hi Christophe, hi Barry, > > I've experimented a bit by including the sc-config switch, as Christophe > mentioned. For me, this results in an aborted process even before the > completion of the first iteration . Now the script fails to run extractor at > ./mert-moses-multi.pl line 1374. I've also attached extract.err.BLEU and TER > which might be helpful. Extract.err.TER says something like "Exception: > std::bad_alloc". My call is > > ./mert-moses-multi.pl > /home/dan/smt/phrase/tuning/devset-1.tok.lw.en > /home/dan/smt/phrase/tuning/devset-1.tok.lw.de > /home/dan/smt/decoder/dist/cb5213a/bin/moses > /home/dan/smt/phrase/model/moses.ini > --working-dir /home/dan/smt/phrase/tuning/mert > --mertdir /home/dan/smt/decoder/mert > --rootdir /home/dan/smt/scripts/cb5213a > --decoder-flags="-v 0 -threads 2 > --sc-config="BLEU:1,TER:3" > > Using WORKING_DIR: /home/dan/smt/phrase/tuning/mert > Using SCRIPTS_ROOTDIR: /home/dan/smt/scripts/cb5213a > filtering the phrase tables... Tue Dec 20 15:37:02 CET 2011 > exec: /home/dan/smt/scripts/cb5213a/training/filter-model-given-input.pl > ./filtered /home/dan/smt/phrase/model/moses.ini > /home/dan/smt/phrase/tuning/m4loc/devset-1.tok.lw.en > Executing: > /home/dan/smt/scripts/cb5213a/training/filter-model-given-input.pl > ./filtered /home/dan/smt/phrase/model/moses.ini > /home/dan/smt/phrase/tuning/devset-1.tok.lw.en> filterphrases.out 2> > filterphrases.err > Asking moses for feature names and values from filtered/moses.ini > Executing: /home/dan/smt/decoder/dist/cb5213a/bin/moses -v 0 -threads 2 > -config filtered/moses.ini -inputtype 0 -show-weights> ./features.list > Creating lexical reordering... > weights: 0.200 0.200 0.200 0.200 0.200 0.200 > Loading table into memory...done. > Creating lexical reordering... > weights: 0.200 0.200 0.200 0.200 0.200 0.200 0.200 0.200 > Loading table into memory...done. > MERT starting values and ranges for random generation: > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > d = 0.200 ( 0.00 .. 1.00) > lm = 0.500 ( 0.00 .. 1.00) > w = -1.000 ( 0.00 .. 1.00) > tm = 0.200 ( 0.00 .. 1.00) > tm = 0.200 ( 0.00 .. 1.00) > tm = 0.200 ( 0.00 .. 1.00) > tm = 0.200 ( 0.00 .. 1.00) > tm = 0.200 ( 0.00 .. 1.00) > run 1 start at Tue Dec 20 15:38:23 CET 2011 > Parsing --decoder-flags: |-v 0 -threads 2| > Saving new config to: ./run1.moses.ini > Saved: ./run1.moses.ini > (1) run decoder to produce n-best lists > params = -v 0 -threads 2 > Normalizing lambdas: 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 > 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 > 0.200000 0.500000 -1.000000 0.200000 0.200000 0.200000 0.200000 0.200000 > DECODER_CFG = -w -0.181818 -lm 0.090909 -d 0.036364 0.036364 0.036364 > 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 > 0.036364 0.036364 0.036364 0.036364 -tm 0.036364 0.036364 0.036364 0.036364 > 0.036364 > decoder_config = -w -0.181818 -lm 0.090909 -d 0.036364 0.036364 0.036364 > 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 > 0.036364 0.036364 0.036364 0.036364 -tm 0.036364 0.036364 0.036364 0.036364 > 0.036364 > Executing: /home/dan/smt/decoder/dist/cb5213a/bin/moses -v 0 -threads 2 > -config filtered/moses.ini -inputtype 0 -w -0.181818 -lm 0.090909 -d > 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 > 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 0.036364 -tm 0.036364 > 0.036364 0.036364 0.036364 0.036364 -n-best-list run1.best100.out 100 > -input-file /home/dan/smt/phrase/tuning/devset-1.tok.lw.en> run1.out > Creating lexical reordering... > weights: 0.036 0.036 0.036 0.036 0.036 0.036 > Loading table into memory...done. > Creating lexical reordering... > weights: 0.036 0.036 0.036 0.036 0.036 0.036 0.036 0.036 > Loading table into memory...done. > Translating line 0 in thread id 3041061744 > reading bin ttable > size of OFF_T 8 > Translating line 1 in thread id 3032669040 > reading bin ttable > size of OFF_T 8 > binary phrasefile loaded, default OFF_T: -1 > binary phrasefile loaded, default OFF_T: -1 > Translating line 2 in thread id 3032669040 > ... > Translating line 999 in thread id 3032669040 > The decoder returns the scores in this order: d d d d d d d d d d d d d d d > lm w tm tm tm tm tm > Executing: gzip -f run1.best100.out > Scoring the nbestlist. > exec: /home/dan/smt/decoder/mert/extractor --scconfig case:true --scfile > run1.scores.dat.BLEU --ffile run1.features.dat.BLEU --sctype BLEU -r > /home/dan/smt/phrase/tuning/devset-1.tok.lw.de -n run1.best100.out.gz > Executing: /home/dan/smt/decoder/mert/extractor --scconfig case:true > --scfile run1.scores.dat.BLEU --ffile run1.features.dat.BLEU --sctype BLEU > -r /home/dan/smt/phrase/tuning/m4loc/devset-1.tok.lw.de -n > run1.best100.out.gz> extract.out.BLEU 2> extract.err.BLEU > exec: /home/dan/smt/decoder/mert/extractor --scconfig case:true --scfile > run1.scores.dat.TER --ffile run1.features.dat.TER --sctype TER -r > /home/dan/smt/phrase/tuning/m4loc/devset-1.tok.lw.de -n run1.best100.out.gz > Executing: /home/dan/smt/decoder/mert/extractor --scconfig case:true > --scfile run1.scores.dat.TER --ffile run1.features.dat.TER --sctype TER -r > /home/dan/smt/phrase/tuning/m4loc/devset-1.tok.lw.de -n run1.best100.out.gz >> extract.out.TER 2> extract.err.TER > Exit code: 1 > ERROR: Failed to run '/home/dan/smt/decoder/mert/extractor --scconfig > case:true --scfile run1.scores.dat.TER --ffile run1.features.dat.TER > --sctype TER -r /home/dan/smt/phrase/tuning/m4loc/devset-1.tok.lw.de -n > run1.best100.out.gz'. at ./mert-moses-multi.pl line 1374. > > Regards, > Daniel > > -----Ursprüngliche Nachricht----- > Von: Daniel Schaut [mailto:[email protected]] > Gesendet: Dienstag, 20. Dezember 2011 15:05 > An: 'Christophe Servan'; '[email protected]' > Cc: 'Barry Haddow' > Betreff: AW: [Moses-support] mert-moses-multi.pl: Failed to run mert at > ./mert-moses-multi.pl line 1374. > > Hi Barry, hi Christophe, > > Thanks for your answers. Please find attached mert.out and mert.log. > >> Why do you think it's running out of memory? > I assume my system ran out of memory, because when it failed to run mert, > memory usage was at 100% for quite a while. Don't know what happened > exactly. I'll try to perform some other runs and keep you updated. > > Regards, > Daniel > > -----Ursprüngliche Nachricht----- > Von: Christophe Servan [mailto:[email protected]] > Gesendet: Montag, 19. Dezember 2011 20:33 > An: [email protected]; Daniel Schaut > Cc: Barry Haddow > Betreff: Re: [Moses-support] mert-moses-multi.pl: Failed to run mert at > ./mert-moses-multi.pl line 1374. > > Hi Daniel, > As Barry said, I made this variation of the mert-moses.pl in order to tune > with multiple metrics together. > The tuning is made with a linear ponderation of metrics, for example : > (1xBLEU+2xTER)/3 > The setting is made with the switch --sc-config="BLEU:1,TER:2" (for my > previous example). > If you don't use this switch, you will tune only with BLEU (the default > metric for tuning). > As Barry proposed, would you like to post the mert.out and mert.log you > generated ? > > Best regards, > > Christophe > > > Le 19/12/2011 15:44, Barry Haddow a écrit : >> Hi Daniel >> >> Why do you think it's running out of memory? Could you post mert.out >> and mert.log ? >> >> Christophe Servan is the person who knows most about this script, >> >> cheers - Barry >> >> On Sunday 18 Dec 2011 18:49:46 Daniel Schaut wrote: >>> mert.out 2> mert.log >>> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
