Problem solved. To review the symptoms, I ran the following two mert-moses-new.pl command lines:
CASE 1: nice mert-moses-new.pl \ /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/mert.en \ /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/mert.nl \ /usr/local/lib/moses-irstlm/moses-cmd/src/moses \ /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/moses0.ini \ --working-dir /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl \ --rootdir /usr/local/lib/moses-irstlm/scripts \ --mertdir=/usr/local/lib/moses-irstlm/mert \ --nbest=50 \ --decoder-flags -v 0 CASE 2: nice mert-moses-new.pl \ /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/mert.en \ /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/mert.nl \ /usr/local/lib/moses-irstlm/moses-cmd/src/moses \ /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/moses0.ini \ --working-dir /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl \ --rootdir /usr/local/lib/moses-irstlm/scripts \ --mertdir=/usr/local/lib/moses-irstlm/mert \ --nbest=50 \ --decoder-flags -v 0 Only one line (the lmodel-file) was different in the respective starting config files: CASE 1: /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/moses0.ini: [ttable-file] 0 0 0 5 /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase-table.gz [lmodel-file] 1 0 3 /media/models/irstlm/europarl.v5.mini/3-gram.nl.blm.mm [distortion-file] 0-0 wbe-msd-bidirectional-fe-allff 6 /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering-table.wbe-msd-bidirectional-fe.gz CASE 2: /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/moses0.ini: [ttable-file] 0 0 0 5 /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase-table.gz [lmodel-file] 1 0 5 /media/models/irstlm/europarl.v5.mini/5-gram.nl.blm.mm [distortion-file] 0-0 wbe-msd-bidirectional-fe-allff 6 /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering-table.wbe-msd-bidirectional-fe.gz In both cases, mert-moses-new.pl filtered the phrase table successfully. In CASE 1, the tuning process continued and concluded with a final moses.ini file with new weights. In CASE 2, however, mert-moses-new.pl created run1.moses.ini. The moses process rapidly (less than 5 minutes) consumed all RAM and virtual memory leaving nothing for other processes. It never sent output to the run1.out file. The system killed moses and mert-moses-new.pl. This occurred from the mert-moses-new.pl script or from the command line using the run1.moses.ini file. Furthermore, I changed run1.moses.ini to use the binarized phrase and reordering tables: 0 0 0 5 /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase-table.gz changed to: 1 0 0 5 /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase-table 0-0 wbe-msd-bidirectional-fe-allff 6 /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering-table.wbe-msd-bidirectional-fe.gz changed to 0-0 wbe-msd-bidirectional-fe-allff 6 /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering-table.wbe-msd-bidirectional-fe With this modified config from the command line (not mert-moses-new.pl), moses loaded in seconds and translated stdin/stdout just fine. Only configurations with the full .gz model and filtered model exhibited the problems. The filtered model, by the way, is only 20 MB for phrase AND reordering tables. SOLUTION: When I first built IRSTLM with MACHTYPE=x86_64, it created $IRSTLM/bin/x86_64. Then, building moses using --with-irstlm=$IRSTLM finished without fatal errors. I recently read the moses-support threads about using a $SRILM/bin/i686 folder. So, I applied the same solution to IRSTLM. I rebuilt IRSTLM and I created two symlinks: ln -s $IRSTLM/bin/x86_64 $IRSTLM/bin/i686 ln -s $IRSTLM/lib/x86_64 $IRSTLM/lib/i686 Then, I rebuild moses --with-irstlm=$IRSTLM. RESULTS: the mert-moses-new.pl script runs flawlessly with 3-gram and 5-gram IRSTLM language models and the exact same config files in CASE 1 and CASE 2 above. Go figure! Hope this helps others. Tom On Wed, 02 Jun 2010 04:31:50 -0700, <[email protected]> wrote: > Thanks. I found a possible problem with the way I built Moses with IRSTLM. > So, I started from scratch and I'm rebuilding phrase tables and language > models. Should be ready for further testing tomorrow. I'll pass my results > when it's done. Stand by... > > Tom > > On Tue, 1 Jun 2010 13:40:35 +0100, Barry Haddow <[email protected]> > wrote: >> Hi Tom >> >> I think 4G ram should be enough for the model you describe, so I don't >> know >> why moses is getting killled. How much memory does it use? Is moses > using >> the >> binarised models? Note that there needs to be a 1 at the start of the >> ttable >> specification for this to happen. eg >> [ttable-file] >> 1 0 0 5 /afs/inf.ed.ac.uk/group/bhaddow/models/fr-en-nc/phrase-table.1 >> >> If you run with '-v 1' then you should be able to see which table is > being >> loaded when the memory exhaustion occurred. >> >> regards >> Barry >> >> On Saturday 29 May 2010 19:23, [email protected] > wrote: >>> I'm troubleshooting a new moses system with these components: >>> 1) GIZA++ (SVN rev 8, v 1.0.3) >>> 2) IRSTLM (SVN rev 38, v 5.40.01) >>> 3) Moses (SVN rev 3210, dated 4-26-2010) >>> 4) Ubuntu-server 10.04 LTS 64-bit. >>> 5) 3.4 Ghz Pentium-D with 4gb ram. >>> >>> Using a 3-gram lm, the system works as expected. Training, tuning and >>> evaluation a small (135K pairs) en-nl subset of europarl.v5 work fine. >>> BLEU >>> score was 23. >>> >>> I then built a 5-gram model, edited the moses.ini config and started >>> mert-moses-new. It creates a filtered model, and then launches moses. > The >>> memory usage grows and within 10 minutes, the system kills moses. >>> >>> In both cases, the lm is only the target half of the bitext corpus, > about >>> 135K lines. >>> >>> The moses.ini files: >>> >>> [lmodel-file] >>> 1 0 3 /media/models/irstlm/europarl.v5.mini/3-gram.nl.blm >>> >>> [lmodel-file] >>> 1 0 5 /media/models/irstlm/europarl.v5.mini/5-gram.nl.blm >>> >>> I know of one other who has anyone the same problem with the 4-1-2010 >>> moses build and irstlm from March/April last year. >>> >>> Any suggestions? Could it be the new Ubuntu or the g++-4.4.1 compiler? >>> >>> Thanks, >>> Tom >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
