also, make sure you compile contrib/sigtest-filter I've added some checks
https://github.com/moses-smt/mosesdecoder/commit/101e52da60a653b3d62dfff75187a657fcf60a94 * Looking for MT/NLP opportunities * Hieu Hoang http://moses-smt.org/ On 2 May 2017 at 10:56, Nikolay Bogoychev <[email protected]> wrote: > Hey Mike, > > Is it possible for you to make the phrase table available > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.txt.gz > publically so we can try to reproduce the problem? > > Cheers, > > Nick > > On Tue, May 2, 2017 at 6:06 AM, Mike Ladwig <[email protected]> wrote: > > Got an exception creating a PT2 with yesterday's master: > > > > Binarize phrase and reordering model in probing table format: > > /home/mike/stelae5/mosesdecoder/scripts/generic/binarize4moses2.perl > > --phrase-table=/home/mike/stelae-projects/de-en/ > phrasemodel/model/phrase-table.gz > > --lex-ro=/home/mike/stelae-projects/de-en/phrasemodel/ > model/reordering-table.wbe-msd-bidirectional-fe.gz > > --output-dir=/home/mike/stelae-projects/de-en/phrasemodel/PT2 > > --num-lex-scores=6 > > Executing: gzip -dc > > /home/mike/stelae-projects/de-en/phrasemodel/model/phrase-table.gz | > > /home/mike/stelae5/mosesdecoder/scripts/generic/. > ./../contrib/sigtest-filter/filter-pt > > -n 0 | gzip -c > > > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.gz > > sh: > > /home/mike/stelae5/mosesdecoder/scripts/generic/. > ./../contrib/sigtest-filter/filter-pt: > > No such file or directory > > Executing: > > /home/mike/stelae5/mosesdecoder/scripts/generic/../../bin/ > processLexicalTableMin > > -in > > /home/mike/stelae-projects/de-en/phrasemodel/model/ > reordering-table.wbe-msd-bidirectional-fe.gz > > -out /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro -T . > > -threads all > > Used options: > > Text reordering table will be read from: > > /home/mike/stelae-projects/de-en/phrasemodel/model/ > reordering-table.wbe-msd-bidirectional-fe.gz > > Output reordering table will be written to: > > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro.minlexr > > Step size for source landmark phrases: 2^10=1024 > > Phrase fingerprint size: 16 bits / P(fp)=1.52588e-05 > > Single Huffman code set for score components: no > > Using score quantization: no > > Running with 24 threads > > > > Pass 1/2: Creating phrase index + Counting scores > > ..................................................[5000000] > > ..................................................[10000000] > > ..................................................[15000000] > > ..................................................[20000000] > > ..................................................[25000000] > > ..................................................[30000000] > > ..................................................[35000000] > > ..................................................[40000000] > > ..................................................[45000000] > > ..................................................[50000000] > > ..................................................[55000000] > > ..................................................[60000000] > > ..................................................[65000000] > > ..................................................[70000000] > > ..................................................[75000000] > > .................... > > > > Intermezzo: Calculating Huffman code sets > > Creating Huffman codes for 32003 scores > > Creating Huffman codes for 16732 scores > > Creating Huffman codes for 31335 scores > > Creating Huffman codes for 32076 scores > > Creating Huffman codes for 15096 scores > > Creating Huffman codes for 31659 scores > > > > Pass 2/2: Compressing scores > > ..................................................[5000000] > > ..................................................[10000000] > > ..................................................[15000000] > > ..................................................[20000000] > > ..................................................[25000000] > > ..................................................[30000000] > > ..................................................[35000000] > > ..................................................[40000000] > > ..................................................[45000000] > > ..................................................[50000000] > > ..................................................[55000000] > > ..................................................[60000000] > > ..................................................[65000000] > > ..................................................[70000000] > > ..................................................[75000000] > > .................... > > > > Saving to > > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro.minlexr > > Done > > Executing: > > /home/mike/stelae5/mosesdecoder/scripts/generic/../../bin/addLexROtoPT > > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.gz > > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro.minlexr | > > gzip -c > > > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.withLexRO.gz > > Executing: ln -s pt.withLexRO.gz > > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.txt.gz > > Executing: > > /home/mike/stelae5/mosesdecoder/scripts/generic/. > ./../bin/CreateProbingPT2 > > --num-scores 4 --log-prob --input-pt > > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.txt.gz > > --output-dir /home/mike/stelae-projects/de-en/phrasemodel/PT2 > > --num-lex-scores 6 > > Starting... > > Reading phrase table finished, writing remaining files to disk. > > terminate called after throwing an instance of > 'util::ProbingSizeException' > > what(): ./util/probing_hash_table.hh:150 in > > util::ProbingHashTable<EntryT, HashT, EqualT, ModT>::Entry* > > util::ProbingHashTable<EntryT, HashT, EqualT, ModT>::Insert(const T&) > [with > > T = probingpt::Entry; EntryT = probingpt::Entry; HashT = boost::hash<long > > unsigned int>; EqualT = std::equal_to<long unsigned int>; ModT = > > util::DivMod; util::ProbingHashTable<EntryT, HashT, EqualT, > > ModT>::MutableIterator = probingpt::Entry*; > util::ProbingHashTable<EntryT, > > HashT, EqualT, ModT>::Entry = probingpt::Entry] threw > ProbingSizeException > > because `++entries_ >= buckets_'. > > Hash table with 1 buckets is full. > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
