Hi Kenneth. Recently I am encountering an increased number of crashes, too. I guess there are some heisenbugs in the binarization that manifest maybe due to a new boost version or something. A workaround is usually to use less threads, only one or up to 4 (it's actually not much faster with 16 anyway). If it still crashes try -encoding None . I am planning to write a new binarization tool from scratch, this one is giving me too much headache.
W dniu 13.01.2015 o 04:20, Kenneth Heafield pisze: > Dear Moses/Marcin, > > I'm getting a Floating point exception in processPhraseTableMin from > Moses d0807c. > > Arguments, minus the absolute paths, are: > > processPhraseTableMin -in phrase-table.gz -out phrase-table -nscores 4 > -threads 16 -T /tmp -encoding None > > The phrase table is rather large and it runs for several hours before > crashing. Log output is below. > > Used options: > Text phrase table will be read from: phrase-table.gz > Output phrase table will be written to: phrase-table.minphr > Step size for source landmark phrases: 2^10=1024 > Source phrase fingerprint size: 16 bits / P(fp)=1.52588e-05 > Selected target phrase encoding: Huffman > Number of score components in phrase table: 4 > Single Huffman code set for score components: no > Using score quantization: no > Explicitly included alignment information: yes > Running with 16 threads > > Pass 1/2: Creating source phrase index + Encoding target phrases > ..................................................[5000000] > ..................................................[10000000] > ..................................................[15000000] > ..................................................[20000000] > ..................................................[25000000] > ..................................................[30000000] > ..................................................[35000000] > ..................................................[40000000] > ..................................................[45000000] > ..................................................[50000000] > ..................................................[55000000] > ..................................................[60000000] > ..................................................[65000000] > ..................................................[70000000] > ..................................................[75000000] > ..................................................[80000000] > ..................................................[85000000] > ..................................................[90000000] > ..................................................[95000000] > ..................................................[100000000] > ..................................................[105000000] > ..................................................[110000000] > ..................................................[115000000] > ..................................................[120000000] > ..................................................[125000000] > ..................................................[130000000] > ..................................................[135000000] > ..................................................[140000000] > ..................................................[145000000] > ..................................................[150000000] > ..................................................[155000000] > ..................................................[160000000] > ..................................................[165000000] > ..................................................[170000000] > ..................................................[175000000] > ..................................................[180000000] > .............................................. > > Intermezzo: Calculating Huffman code sets > Creating Huffman codes for 624564 target phrase symbols > Creating Huffman codes for 551381 scores > Creating Huffman codes for 15296482 scores > Creating Huffman codes for 582875 scores > Creating Huffman codes for 15806633 scores > Creating Huffman codes for 50 alignment points > > Pass 2/2: Compressing target phrases > ..................................................[5000000] > ..................................................[10000000] > > Kenneth > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
