Hi, Eric, Eric Parker wrote: > # translation tables: source-factors, target-factors, number of scores, file > [ttable-file] > 1 0 0 5 /home/mwade/demo/work2/model/phrase-table
the first number indicates the type of the phrase table, see the enum PhraseTableImplementation in moses/src/TypeDef.h the second are the source factors (factor 0 only in your case) the third are target factors the fourth is the number of weights/scores in the phrase table Similarly for the lmodel-file. > Everything else in the file is default, but I can post the entire > thing if needed. Sometimes, it may be also useful to post some first few lines of each model file (when still in plain text format). > I ran the sanity check and it passed. What sanity check? However when I run the following command: > nohup nice tools/moses-scripts/scripts-20100503-1638/training/mert-moses.pl ... > The machine runs out of memory. Here is the relevant info from mert.out: > > filtering > /home/mwade/demo/work/model/reordering-table.wbe-msd-bidirectional-fe > -> > /home/mwade/demo/work/tuning/mert/filtered/reordering-table.wbe-msd-bidirectional-fe... > Can't open 'gzip -cd > /home/mwade/demo/work/model/reordering-table.wbe-msd-bidirectional-fe.gz > |' at > /home/mwade/demo/tools/moses-scripts/scripts-20100503-1638//training/filter-model-given-input.pl > line 222. > Exit code: 12 > Failed to filter the tables. at > tools/moses-scripts/scripts-20100503-1638/training/mert-moses.pl line > 491. > > > It doesn't look like it is using the binary version of the > reordering-table to me. The file it says it can not open does exist > and I am able to open it fine by hand. I ran the command using the > 'head -n 100' subset and it worked fine, so everything else appears to > be working properly. I just can't get it to run all the way through > without running out of memory on the full file set. I ran an strace > on the command that showed my memory issue: > > clone(child_stack=0, > flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, > child_tidptr=0x2b7986ec3e20) = -1 ENOMEM (Cannot allocate memory) I see! This is a known issue of unix: if your process has already grown beyond 50% of memory and it tries to fork (to exec the gunzip in the child process afterwards), the fork usually fails, because it first needs to duplicate the whole huge process. I'm surprised to see filter-model-given-input.perl to allocate much memory though, I guess this is related to some new things in the chart/hierarchical models and I'm not familiar with that yet. Cheers, Ondrej. > > > I know this is a lot of data to go through, and I hope I provided > enough information for someone to help me out. If anything else is > needed, please let me know. I really need to find a solution to this, > and I am sure it is a simple fix, but I can't find it. Thanks in > advance! > > ----- > Eric Parker > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support -- Ondrej Bojar (mailto:[email protected] / [email protected]) http://www.cuni.cz/~obo _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
