i'm not an expert on irstlm but i think you have to rename or softlink
the file so that the extension ends in .mm to minimise memory usage.
http://www.statmt.org/moses/?n=FactoredTraining.BuildingLanguageModel#ntoc7
otherwise it'll just allocate as much memory as needed to load the LM file
On 29/05/2010 19:23, [email protected] wrote:
> I'm troubleshooting a new moses system with these components:
> 1) GIZA++ (SVN rev 8, v 1.0.3)
> 2) IRSTLM (SVN rev 38, v 5.40.01)
> 3) Moses (SVN rev 3210, dated 4-26-2010)
> 4) Ubuntu-server 10.04 LTS 64-bit.
> 5) 3.4 Ghz Pentium-D with 4gb ram.
>
> Using a 3-gram lm, the system works as expected. Training, tuning and
> evaluation a small (135K pairs) en-nl subset of europarl.v5 work fine. BLEU
> score was 23.
>
> I then built a 5-gram model, edited the moses.ini config and started
> mert-moses-new. It creates a filtered model, and then launches moses. The
> memory usage grows and within 10 minutes, the system kills moses.
>
> In both cases, the lm is only the target half of the bitext corpus, about
> 135K lines.
>
> The moses.ini files:
>
> [lmodel-file]
> 1 0 3 /media/models/irstlm/europarl.v5.mini/3-gram.nl.blm
>
> [lmodel-file]
> 1 0 5 /media/models/irstlm/europarl.v5.mini/5-gram.nl.blm
>
> I know of one other who has anyone the same problem with the 4-1-2010
> moses build and irstlm from March/April last year.
>
> Any suggestions? Could it be the new Ubuntu or the g++-4.4.1 compiler?
>
> Thanks,
> Tom
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
