you're running out of memory. Binarize your phrase table
On 09/09/2013 22:34, rohit dholakia wrote:
Hi,
Alright, my phrase table is just 100MB compressed. And after taking
quite long to read the phrase table, i got an exception bad_alloc. I
have quite a bit of RAM, so, i am not sure what I am doing wrong .
Start loading text SCFG phrase table. Moses format : [51.000] seconds
Reading /global/scratch/rd/Thesis/TestingMoses/train/model/phrase-table.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Exception: std::bad_alloc
Are the changes in the master branch mostly related to an updated
moses file format ? Are there any changes in the algorithms(mert/ibm
models) that warrant a re-run of tests done using the previous version ?
On Mon, Sep 9, 2013 at 1:24 PM, rohit dholakia <[email protected]
<mailto:[email protected]>> wrote:
Hi
I have recently managed to install moses from the master branch
on github. The file format has changed. The training and mert has
gone without incident. But, i have observed that reading the
phrase table is taking very very long compared to the previous
moses. From mert.out and from translating test data log,
/Loading table into memory...done./
/Start loading text SCFG phrase table. Moses format : [51.000]
seconds/
/Reading
/global/scratch/rd/Thesis/TestingMoses/train/model/phrase-table.gz/
/----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100/
Now, this step, this step is taking really long. Am I the only one
who is observing this ? My phrase table is trained on only 110K
sentences, quite noisy ones at that. Has only 4.8 million rules.
/[rd@seawolf1 TestingMoses]$ zcat train/model/phrase-table.gz | wc -l/
/4838810/
To explain the time taken, its been more than 5 minutes and not
even a single star has come up below that last line ( ---- 5 ---
10 ) .
Thanks.
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
