hi Tom I supposed there is a mismatch with the last working IRSTLM release and your.
Where did you get yours? Nicola On Jun 9, 2010, at 6:36 AM, <[email protected]> wrote: > "Problem solved..." I spoke too early. > > Creating the symlink, $SRILM/bin/i686, enabled me to run MERT with > small > (135K pair) test sets with a 5-gram irstlm whereas before they crashed > immediately. However, it failed when I move to a training corpus > with 1.3 > million pairs. The moses-irstlm system ran for about 1 1/2 days and > only > output 939 of 3842 phrases in the run1. During that time, it gradually > consuming RAM and swap space. It finally failed with the following > fatal > message: > > [113615.909787] Out of Memory : kill process 299 (sh) score > 53126 or a > child [113615.909826] Killed process 2100 (moses) > > I just finished successfully tuning the exact same phrase/reordering > tables with an srilm model. The srilm model was built with the > exact same > data as the irstlm model that fails. Attachments include: > > 1) irstlm.europarl.v5.mini.en-nl.tar.gz -- the successful srilm > logs/configs > 2) irstlm.europarl.v5.en-nl.tar.gz -- the failed irstlm logs/configs > 3) srilm.europarl.v5.en-nl.tar.gz -- the successful srilm logs/configs > using the same tables > > Linux 'top' reported that during mert with moses-srilm, the moses > instance > consumed about 3.5 GB of RAM with practically zero swap file usage. > This > held constant through each iteration. The irstlm version, however, > gradually consumes all RAM and then it consumed the swap file before > crashing. > > To review the builds: > 1) GIZA++ (SVN rev 8, v 1.0.3) > 2) IRSTLM (SVN rev 38, v 5.40.01) > 3) SRILM (ver 1.5.10) > 4) Moses (SVN rev 3210, dated 4-26-2010) > 5) Ubuntu-server 10.04 LTS 64-bit > 6) 2.6 Ghz Core2-Quad with 4GB RAM > > Have any others had similar problems? There's probably a simple > solution, > but I haven't found it. I configured the .ini file for irstlm to > use the > .mm memory-mapped binarized model. Can someone review the attached > log and > moses.ini files? > > Thanks. > > Tom > > > On Wed, 02 Jun 2010 21:48:33 -0700, > <[email protected]> > wrote: >> Problem solved. >> >> To review the symptoms, I ran the following two mert-moses-new.pl > command >> lines: >> >> CASE 1: >> nice mert-moses-new.pl \ >> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/ >> mert.en > \ >> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/ >> mert.nl > \ >> /usr/local/lib/moses-irstlm/moses-cmd/src/moses \ >> >> > /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/ > moses0.ini > \ >> --working-dir > /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl \ >> --rootdir /usr/local/lib/moses-irstlm/scripts \ >> --mertdir=/usr/local/lib/moses-irstlm/mert \ >> --nbest=50 \ >> --decoder-flags -v 0 >> >> CASE 2: >> nice mert-moses-new.pl \ >> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/ >> mert.en > \ >> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/ >> mert.nl > \ >> /usr/local/lib/moses-irstlm/moses-cmd/src/moses \ >> > /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/ > moses0.ini > \ >> --working-dir > /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl \ >> --rootdir /usr/local/lib/moses-irstlm/scripts \ >> --mertdir=/usr/local/lib/moses-irstlm/mert \ >> --nbest=50 \ >> --decoder-flags -v 0 >> >> >> Only one line (the lmodel-file) was different in the respective >> starting >> config files: >> >> CASE 1: >> > /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/ > moses0.ini: >> [ttable-file] >> 0 0 0 5 >> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase- >> table.gz >> [lmodel-file] >> 1 0 3 /media/models/irstlm/europarl.v5.mini/3-gram.nl.blm.mm >> [distortion-file] >> 0-0 wbe-msd-bidirectional-fe-allff 6 >> > /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering- > table.wbe-msd-bidirectional-fe.gz >> >> CASE 2: >> > /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/ > moses0.ini: >> [ttable-file] >> 0 0 0 5 >> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase- >> table.gz >> [lmodel-file] >> 1 0 5 /media/models/irstlm/europarl.v5.mini/5-gram.nl.blm.mm >> [distortion-file] >> 0-0 wbe-msd-bidirectional-fe-allff 6 >> > /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering- > table.wbe-msd-bidirectional-fe.gz >> >> >> In both cases, mert-moses-new.pl filtered the phrase table >> successfully. >> In CASE 1, the tuning process continued and concluded with a final >> moses.ini file with new weights. In CASE 2, however, mert-moses- >> new.pl >> created run1.moses.ini. The moses process rapidly (less than 5 >> minutes) >> consumed all RAM and virtual memory leaving nothing for other >> processes. > It >> never sent output to the run1.out file. The system killed moses and >> mert-moses-new.pl. This occurred from the mert-moses-new.pl script or > from >> the command line using the run1.moses.ini file. >> >> Furthermore, I changed run1.moses.ini to use the binarized phrase and >> reordering tables: >> 0 0 0 5 >> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase- >> table.gz >> changed to: >> 1 0 0 5 >> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase-table >> >> 0-0 wbe-msd-bidirectional-fe-allff 6 >> > /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering- > table.wbe-msd-bidirectional-fe.gz >> changed to >> 0-0 wbe-msd-bidirectional-fe-allff 6 >> > /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering- > table.wbe-msd-bidirectional-fe >> >> With this modified config from the command line (not mert-moses- >> new.pl), >> moses loaded in seconds and translated stdin/stdout just fine. Only >> configurations with the full .gz model and filtered model >> exhibited the >> problems. The filtered model, by the way, is only 20 MB for phrase >> AND >> reordering tables. >> >> >> SOLUTION: >> When I first built IRSTLM with MACHTYPE=x86_64, it created >> $IRSTLM/bin/x86_64. Then, building moses using --with-irstlm=$IRSTLM >> finished without fatal errors. I recently read the moses-support >> threads >> about using a $SRILM/bin/i686 folder. So, I applied the same >> solution to >> IRSTLM. I rebuilt IRSTLM and I created two symlinks: >> ln -s $IRSTLM/bin/x86_64 $IRSTLM/bin/i686 >> ln -s $IRSTLM/lib/x86_64 $IRSTLM/lib/i686 >> >> Then, I rebuild moses --with-irstlm=$IRSTLM. >> >> RESULTS: the mert-moses-new.pl script runs flawlessly with 3-gram and >> 5-gram IRSTLM language models and the exact same config files in >> CASE 1 > and >> CASE 2 above. >> >> Go figure! >> >> Hope this helps others. >> >> Tom >> >> >> On Wed, 02 Jun 2010 04:31:50 -0700, >> <[email protected]> >> wrote: >>> Thanks. I found a possible problem with the way I built Moses with >> IRSTLM. >>> So, I started from scratch and I'm rebuilding phrase tables and > language >>> models. Should be ready for further testing tomorrow. I'll pass my >> results >>> when it's done. Stand by... >>> >>> Tom >>> >>> On Tue, 1 Jun 2010 13:40:35 +0100, Barry Haddow >>> <[email protected]> >>> wrote: >>>> Hi Tom >>>> >>>> I think 4G ram should be enough for the model you describe, so I >>>> don't >>>> know >>>> why moses is getting killled. How much memory does it use? Is moses >>> using >>>> the >>>> binarised models? Note that there needs to be a 1 at the start >>>> of the >>>> ttable >>>> specification for this to happen. eg >>>> [ttable-file] >>>> 1 0 0 5 /afs/inf.ed.ac.uk/group/bhaddow/models/fr-en-nc/phrase- >>>> table.1 >>>> >>>> If you run with '-v 1' then you should be able to see which >>>> table is >>> being >>>> loaded when the memory exhaustion occurred. >>>> >>>> regards >>>> Barry >>>> >>>> On Saturday 29 May 2010 19:23, >>>> [email protected] >>> wrote: >>>>> I'm troubleshooting a new moses system with these components: >>>>> 1) GIZA++ (SVN rev 8, v 1.0.3) >>>>> 2) IRSTLM (SVN rev 38, v 5.40.01) >>>>> 3) Moses (SVN rev 3210, dated 4-26-2010) >>>>> 4) Ubuntu-server 10.04 LTS 64-bit. >>>>> 5) 3.4 Ghz Pentium-D with 4gb ram. >>>>> >>>>> Using a 3-gram lm, the system works as expected. Training, >>>>> tuning and >>>>> evaluation a small (135K pairs) en-nl subset of europarl.v5 work > fine. >>>>> BLEU >>>>> score was 23. >>>>> >>>>> I then built a 5-gram model, edited the moses.ini config and >>>>> started >>>>> mert-moses-new. It creates a filtered model, and then launches >>>>> moses. >>> The >>>>> memory usage grows and within 10 minutes, the system kills moses. >>>>> >>>>> In both cases, the lm is only the target half of the bitext >>>>> corpus, >>> about >>>>> 135K lines. >>>>> >>>>> The moses.ini files: >>>>> >>>>> [lmodel-file] >>>>> 1 0 3 /media/models/irstlm/europarl.v5.mini/3-gram.nl.blm >>>>> >>>>> [lmodel-file] >>>>> 1 0 5 /media/models/irstlm/europarl.v5.mini/5-gram.nl.blm >>>>> >>>>> I know of one other who has anyone the same problem with the >>>>> 4-1-2010 >>>>> moses build and irstlm from March/April last year. >>>>> >>>>> Any suggestions? Could it be the new Ubuntu or the g++-4.4.1 > compiler? >>>>> >>>>> Thanks, >>>>> Tom >>>>> _______________________________________________ >>>>> Moses-support mailing list >>>>> [email protected] >>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses- >> support<irstlm.europarl.v5.mini.en- >> nl.tar.gz><irstlm.europarl.v5.en-nl.tar.gz><srilm.europarl.v5.en- >> nl.tar.gz> _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
