hiya tom, i used the latest version of binarizing lexical reordering recently & didn't find anything wrong with it.
the 1st 40 or so lines in your data seems to have weird characters which is making the binarizer go into a tailspin. When I run it on the last 900 lines, its ok On 27 August 2010 05:14, <[email protected]> wrote: > I've trained a new model and preparing to run mert. > > 1) train-factored-phrase-model.perl completed without errors (attached > output_log.tar.gz file) > > 2) processPhraseTable completed without errors and reported: > > ..........phrase: 14500000 > ...distinct source phrases: 14553617 distinct first words of source phrase: > 79372 number of phrase pairs (line count): 25544558 > WARNING: there are src voc entries with no phrase translation: count 5243 > There exists phrase translations for 74129 entries > > 3) processLexicalTable failed with the following details: > > $ gzip -cd > /project10/model.en-nl/reordering-table.wbe-msd-bidirectional-fe.gz | sort | > processLexicalTable -out > /project10/model.en-nl/reordering-table.wbe-msd-bidirectional-fe > processLexicalTable v0.1 by Konrad Rawlik > processing stdin to > /project10/model.en-nl/reordering-table.wbe-msd-bidirectional-fe.* > terminate called after throwing an instance of 'std::bad_alloc' > what(): std::bad_alloc > Aborted > > > > The reordering-table.wbe-msd-bidirectional-fe.gz file is 252 MB. I also > saved sorted/expanded data (2.5 GB) and ran processLexicalTable. It > terminates almost immediately with the same error. > > I attached a zip file with the train-factored-phrase-model.perl log file, > two binary files created by processLexicalTable and the first 10,000 lines > of the sorted phrase table. Is this a data problem? > > Thanks, > > Tom > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
