Hi Nicola, Any progress or suggestions for getting IRSTLM working with moses in this configuration: 1) IRSTLM v5.40.01 (svn revision 388) 2) moses dated 4.26.2010 (svn revision 3210) 3) .ini configs and logs attached to previous message 4) etc. (below)
I neglected to mention in previous messages, that the moses with 5-gram irstlm works fine when configured to use binarized phrase/reorder tables. It only consumes memory when configured to use filtered tables. Same filtered tables work fine with the srilm version of moses. Could it be an .ini config problem, or should I use another moses (or irstlm) build? Thanks, Tom On Wed, 09 Jun 2010 02:54:20 -0700, <[email protected]> wrote: > Thanks Nicola, > > I download the latest IRSTLM directly from SVN with: > > svn co -r 388 https://irstlm.svn.sourceforge.net/svnroot/irstlm/ > ./irstlm > (oops, message below stated 38... typo) > > Then I patch lmtabe.h file: > > patch -p0 < irstlm-rev388-lmtable.patch > > CONTENTS irstlm-rev388-lmtable.patch: > ********************************************************** > --- /usr/local/src/irstlm/src/lmtable.h (revision 388) > +++ /usr/local/src/irstlm/src/lmtable.h (working copy) > @@ -29,6 +29,7 @@ > #include <sys/mman.h> > #endif > > +#include <limits> > #include <math.h> > #include <cstdlib> > ********************************************************** > > Then, I run the commands: > ./regenerate-makefiles.sh > ./configure --prefix=/usr/local/src/irstlm > make > make install > > > A colleague across the world shares the same problem. His email said: > "Note that I had the problem also with IRSTLM 5.22.01 (with Moses > 4/26/2010 or 4/01/2010), which was working fine with last year’s Moses > release." > > This message from Ondrej Bojar on 06/05/2010 talked about memory growth on > 'unix' platforms. Could this be a problem here? > > From: Ondrej Bojar > Date: 06.05.2010 21:35 > Subject: Re: [Moses-support] Need help with Part IV - Tuning > > This is a known issue of unix: if your process has already grown > beyond 50% of memory and it tries to fork (to exec the gunzip in > the child process afterwards), the fork usually fails, because > it first needs to duplicate the whole huge process. > > > Thanks in advance. > Tom > > > > On Wed, 9 Jun 2010 09:41:21 +0200, Nicola Bertoldi <[email protected]> > wrote: >> hi Tom >> >> I supposed there is a mismatch with the last working IRSTLM release >> and your. >> >> Where did you get yours? >> >> Nicola >> >> On Jun 9, 2010, at 6:36 AM, <[email protected]> >> wrote: >> >>> "Problem solved..." I spoke too early. >>> >>> Creating the symlink, $SRILM/bin/i686, enabled me to run MERT with >>> small >>> (135K pair) test sets with a 5-gram irstlm whereas before they crashed >>> immediately. However, it failed when I move to a training corpus >>> with 1.3 >>> million pairs. The moses-irstlm system ran for about 1 1/2 days and >>> only >>> output 939 of 3842 phrases in the run1. During that time, it gradually >>> consuming RAM and swap space. It finally failed with the following >>> fatal >>> message: >>> >>> [113615.909787] Out of Memory : kill process 299 (sh) score >>> 53126 or a >>> child [113615.909826] Killed process 2100 (moses) >>> >>> I just finished successfully tuning the exact same phrase/reordering >>> tables with an srilm model. The srilm model was built with the >>> exact same >>> data as the irstlm model that fails. Attachments include: >>> >>> 1) irstlm.europarl.v5.mini.en-nl.tar.gz -- the successful srilm >>> logs/configs >>> 2) irstlm.europarl.v5.en-nl.tar.gz -- the failed irstlm logs/configs >>> 3) srilm.europarl.v5.en-nl.tar.gz -- the successful srilm logs/configs >>> using the same tables >>> >>> Linux 'top' reported that during mert with moses-srilm, the moses >>> instance >>> consumed about 3.5 GB of RAM with practically zero swap file usage. >>> This >>> held constant through each iteration. The irstlm version, however, >>> gradually consumes all RAM and then it consumed the swap file before >>> crashing. >>> >>> To review the builds: >>> 1) GIZA++ (SVN rev 8, v 1.0.3) >>> 2) IRSTLM (SVN rev 38, v 5.40.01) >>> 3) SRILM (ver 1.5.10) >>> 4) Moses (SVN rev 3210, dated 4-26-2010) >>> 5) Ubuntu-server 10.04 LTS 64-bit >>> 6) 2.6 Ghz Core2-Quad with 4GB RAM >>> >>> Have any others had similar problems? There's probably a simple >>> solution, >>> but I haven't found it. I configured the .ini file for irstlm to >>> use the >>> .mm memory-mapped binarized model. Can someone review the attached >>> log and >>> moses.ini files? >>> >>> Thanks. >>> >>> Tom >>> >>> >>> On Wed, 02 Jun 2010 21:48:33 -0700, >>> <[email protected]> >>> wrote: >>>> Problem solved. >>>> >>>> To review the symptoms, I ran the following two mert-moses-new.pl >>> command >>>> lines: >>>> >>>> CASE 1: >>>> nice mert-moses-new.pl \ >>>> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/ >>>> mert.en >>> \ >>>> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/ >>>> mert.nl >>> \ >>>> /usr/local/lib/moses-irstlm/moses-cmd/src/moses \ >>>> >>>> >>> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/ >>> moses0.ini >>> \ >>>> --working-dir >>> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl \ >>>> --rootdir /usr/local/lib/moses-irstlm/scripts \ >>>> --mertdir=/usr/local/lib/moses-irstlm/mert \ >>>> --nbest=50 \ >>>> --decoder-flags -v 0 >>>> >>>> CASE 2: >>>> nice mert-moses-new.pl \ >>>> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/ >>>> mert.en >>> \ >>>> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/ >>>> mert.nl >>> \ >>>> /usr/local/lib/moses-irstlm/moses-cmd/src/moses \ >>>> >>> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/ >>> moses0.ini >>> \ >>>> --working-dir >>> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl \ >>>> --rootdir /usr/local/lib/moses-irstlm/scripts \ >>>> --mertdir=/usr/local/lib/moses-irstlm/mert \ >>>> --nbest=50 \ >>>> --decoder-flags -v 0 >>>> >>>> >>>> Only one line (the lmodel-file) was different in the respective >>>> starting >>>> config files: >>>> >>>> CASE 1: >>>> >>> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/ >>> moses0.ini: >>>> [ttable-file] >>>> 0 0 0 5 >>>> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase- >>>> table.gz >>>> [lmodel-file] >>>> 1 0 3 /media/models/irstlm/europarl.v5.mini/3-gram.nl.blm.mm >>>> [distortion-file] >>>> 0-0 wbe-msd-bidirectional-fe-allff 6 >>>> >>> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering- >>> table.wbe-msd-bidirectional-fe.gz >>>> >>>> CASE 2: >>>> >>> /media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/ >>> moses0.ini: >>>> [ttable-file] >>>> 0 0 0 5 >>>> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase- >>>> table.gz >>>> [lmodel-file] >>>> 1 0 5 /media/models/irstlm/europarl.v5.mini/5-gram.nl.blm.mm >>>> [distortion-file] >>>> 0-0 wbe-msd-bidirectional-fe-allff 6 >>>> >>> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering- >>> table.wbe-msd-bidirectional-fe.gz >>>> >>>> >>>> In both cases, mert-moses-new.pl filtered the phrase table >>>> successfully. >>>> In CASE 1, the tuning process continued and concluded with a final >>>> moses.ini file with new weights. In CASE 2, however, mert-moses- >>>> new.pl >>>> created run1.moses.ini. The moses process rapidly (less than 5 >>>> minutes) >>>> consumed all RAM and virtual memory leaving nothing for other >>>> processes. >>> It >>>> never sent output to the run1.out file. The system killed moses and >>>> mert-moses-new.pl. This occurred from the mert-moses-new.pl script or >>> from >>>> the command line using the run1.moses.ini file. >>>> >>>> Furthermore, I changed run1.moses.ini to use the binarized phrase and >>>> reordering tables: >>>> 0 0 0 5 >>>> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase- >>>> table.gz >>>> changed to: >>>> 1 0 0 5 >>>> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase-table >>>> >>>> 0-0 wbe-msd-bidirectional-fe-allff 6 >>>> >>> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering- >>> table.wbe-msd-bidirectional-fe.gz >>>> changed to >>>> 0-0 wbe-msd-bidirectional-fe-allff 6 >>>> >>> /media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering- >>> table.wbe-msd-bidirectional-fe >>>> >>>> With this modified config from the command line (not mert-moses- >>>> new.pl), >>>> moses loaded in seconds and translated stdin/stdout just fine. Only >>>> configurations with the full .gz model and filtered model >>>> exhibited the >>>> problems. The filtered model, by the way, is only 20 MB for phrase >>>> AND >>>> reordering tables. >>>> >>>> >>>> SOLUTION: >>>> When I first built IRSTLM with MACHTYPE=x86_64, it created >>>> $IRSTLM/bin/x86_64. Then, building moses using --with-irstlm=$IRSTLM >>>> finished without fatal errors. I recently read the moses-support >>>> threads >>>> about using a $SRILM/bin/i686 folder. So, I applied the same >>>> solution to >>>> IRSTLM. I rebuilt IRSTLM and I created two symlinks: >>>> ln -s $IRSTLM/bin/x86_64 $IRSTLM/bin/i686 >>>> ln -s $IRSTLM/lib/x86_64 $IRSTLM/lib/i686 >>>> >>>> Then, I rebuild moses --with-irstlm=$IRSTLM. >>>> >>>> RESULTS: the mert-moses-new.pl script runs flawlessly with 3-gram and >>>> 5-gram IRSTLM language models and the exact same config files in >>>> CASE 1 >>> and >>>> CASE 2 above. >>>> >>>> Go figure! >>>> >>>> Hope this helps others. >>>> >>>> Tom >>>> >>>> >>>> On Wed, 02 Jun 2010 04:31:50 -0700, >>>> <[email protected]> >>>> wrote: >>>>> Thanks. I found a possible problem with the way I built Moses with >>>> IRSTLM. >>>>> So, I started from scratch and I'm rebuilding phrase tables and >>> language >>>>> models. Should be ready for further testing tomorrow. I'll pass my >>>> results >>>>> when it's done. Stand by... >>>>> >>>>> Tom >>>>> >>>>> On Tue, 1 Jun 2010 13:40:35 +0100, Barry Haddow >>>>> <[email protected]> >>>>> wrote: >>>>>> Hi Tom >>>>>> >>>>>> I think 4G ram should be enough for the model you describe, so I >>>>>> don't >>>>>> know >>>>>> why moses is getting killled. How much memory does it use? Is moses >>>>> using >>>>>> the >>>>>> binarised models? Note that there needs to be a 1 at the start >>>>>> of the >>>>>> ttable >>>>>> specification for this to happen. eg >>>>>> [ttable-file] >>>>>> 1 0 0 5 /afs/inf.ed.ac.uk/group/bhaddow/models/fr-en-nc/phrase- >>>>>> table.1 >>>>>> >>>>>> If you run with '-v 1' then you should be able to see which >>>>>> table is >>>>> being >>>>>> loaded when the memory exhaustion occurred. >>>>>> >>>>>> regards >>>>>> Barry >>>>>> >>>>>> On Saturday 29 May 2010 19:23, >>>>>> [email protected] >>>>> wrote: >>>>>>> I'm troubleshooting a new moses system with these components: >>>>>>> 1) GIZA++ (SVN rev 8, v 1.0.3) >>>>>>> 2) IRSTLM (SVN rev 38, v 5.40.01) >>>>>>> 3) Moses (SVN rev 3210, dated 4-26-2010) >>>>>>> 4) Ubuntu-server 10.04 LTS 64-bit. >>>>>>> 5) 3.4 Ghz Pentium-D with 4gb ram. >>>>>>> >>>>>>> Using a 3-gram lm, the system works as expected. Training, >>>>>>> tuning and >>>>>>> evaluation a small (135K pairs) en-nl subset of europarl.v5 work >>> fine. >>>>>>> BLEU >>>>>>> score was 23. >>>>>>> >>>>>>> I then built a 5-gram model, edited the moses.ini config and >>>>>>> started >>>>>>> mert-moses-new. It creates a filtered model, and then launches >>>>>>> moses. >>>>> The >>>>>>> memory usage grows and within 10 minutes, the system kills moses. >>>>>>> >>>>>>> In both cases, the lm is only the target half of the bitext >>>>>>> corpus, >>>>> about >>>>>>> 135K lines. >>>>>>> >>>>>>> The moses.ini files: >>>>>>> >>>>>>> [lmodel-file] >>>>>>> 1 0 3 /media/models/irstlm/europarl.v5.mini/3-gram.nl.blm >>>>>>> >>>>>>> [lmodel-file] >>>>>>> 1 0 5 /media/models/irstlm/europarl.v5.mini/5-gram.nl.blm >>>>>>> >>>>>>> I know of one other who has anyone the same problem with the >>>>>>> 4-1-2010 >>>>>>> moses build and irstlm from March/April last year. >>>>>>> >>>>>>> Any suggestions? Could it be the new Ubuntu or the g++-4.4.1 >>> compiler? >>>>>>> >>>>>>> Thanks, >>>>>>> Tom >>>>>>> _______________________________________________ >>>>>>> Moses-support mailing list >>>>>>> [email protected] >>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>> _______________________________________________ >>>>> Moses-support mailing list >>>>> [email protected] >>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> _______________________________________________ >>>> Moses-support mailing list >>>> [email protected] >>>> http://mailman.mit.edu/mailman/listinfo/moses- >>>> support<irstlm.europarl.v5.mini.en- >>>> nl.tar.gz><irstlm.europarl.v5.en-nl.tar.gz><srilm.europarl.v5.en- >>>> nl.tar.gz> > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
