Ooh right, didn't look at that. That's weird. Is there something special about that phrase table other than the 7 scores? It seems to build a huge huffman tree for the the first set of scores. Based on the reported memory allocation it seems to be quite big, but not big enough that it should cause any problems.
On 02.02.2016 16:58, Kenneth Heafield wrote: > That typically causes a bus error. Why is there an overly huge malloc? > > On 02/02/2016 03:53 PM, Marcin Junczys-Dowmunt wrote: >> I think it fills up your temporary folder, try "-T ." to specify thew >> local folder for temporary files. >> >> On 02.02.2016 16:21, Jeremy Gwinnup wrote: >>> Hi, >>> >>> I’m having a problem using processPhraseTableMin to compress a phrase table >>> with 7 scores - the program consistently coredumps at step 3 - command and >>> relevant output below. Is there anything I’m doing glaringly wrong? >>> >>> Thanks! >>> -Jeremy >>> >>> Command: >>> >>> 1tqoct1:model> $MOSES/bin/processPhraseTableMin -in phrase-table.1.gz -out >>> phrase-table.1 -threads all -nscores 7 >>> >>> Once we get to step 3: >>> >>> Intermezzo: Calculating Huffman code sets >>> Creating Huffman codes for 471366 target phrase symbols >>> tcmalloc: large alloc 13983629312 bytes == 0xb14ce000 @ >>> tcmalloc: large alloc 27967250432 bytes == 0x3f3ca4000 @ >>> tcmalloc: large alloc 15681406635450368 bytes == (nil) @ >>> terminate called after throwing an instance of 'std::bad_alloc' >>> what(): std::bad_alloc >>> >>> Top looked like this when the program ran into trouble: >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>> 27416 jgwinnup 20 0 45.9g 30g 4.0g R 10.6 3.0 1589:17 processPhraseTa >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
