Hi, I think it should work. Does this have something to do with available disk space?
2017-03-29 23:40, Kenneth Heafield: > How embarrassing. Can you try on head from github.com/kpu/kenlm > <http://github.com/kpu/kenlm> ? If that fails, I can take this off list. > > Kenneth > > On March 29, 2017 3:39:20 PM GMT+01:00, Dingyuan Wang > <[email protected]> wrote: > > Dear list, > > lmplz crashed on my machine recently. Command is > > lmplz -o 4 -S 70% --text zhc-simp.txt --arpa zhc.lm --prune 0 1 1 2 > > === 1/5 Counting and sorting n-grams === > Reading /home/gumble/docs/E/corpus/zhs/zhc-simp.txt > > ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 > tcmalloc: large alloc 2340552704 bytes == 0x55e7ed4f4000 @ > tcmalloc: large alloc 9362194432 bytes == 0x55e878d14000 @ > > **************************************************************************************************** > Unigram tokens 886453003 types 66249 > === 2/5 Calculating and sorting adjusted counts === > Chain sizes: 1:794988 2:1961835648 3:3678441728 4:5885507072 > tcmalloc: large alloc 5885509632 bytes == 0x55e7ed4f4000 @ > tcmalloc: large alloc 1961836544 bytes == 0x55e94c29c000 @ > tcmalloc: large alloc 3678445568 bytes == 0x55e9c1190000 @ > Statistics: > 1 66249 D1=0.549028 D2=1.18255 D3+=0.99644 > 2 14266408/22790840 D1=0.615082 D2=1.06095 D3+=1.47555 > 3 87810872/205978808 D1=0.742285 D2=1.17282 D3+=1.49899 > 4 62909089/415283792 D1=0.698985 D2=1.20588 D3+=1.54463 > Memory estimate for binary LM: > type MB > probing 3417 assuming -p 1.5 > probing 4002 assuming -r models -p 1.5 > trie 1653 without quantization > trie 908 assuming -q 8 -b 8 quantization > trie 1418 assuming -a 22 array pointer compression > trie 674 assuming -a 22 -q 8 -b 8 array pointer compression and > quantization > === 3/5 Calculating and sorting initial probabilities === > tcmalloc: large alloc 4119576576 bytes == 0x55e94c1d8000 @ > tcmalloc: large alloc 9966813184 bytes == 0x55eaaf630000 @ > Chain sizes: 1:794988 2:228262528 3:1756217440 4:1509818136 > > ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 > > ##**********###############################################################-----##**********++#############################################################-----##************#############################################################-----##************####################################################################************####################################################################************+###################################################################*************###################################################################*************##################################################################################### > === 4/5 Calculating and writing order-interpolated probabilities === > Chain sizes: 1:794988 2:228262528 3:1756217440 4:1509818136 > > ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 > ------------------------------------------------------------------------ > terminate > called after throwing an instance of 'lm::FormatLoadException' > what(): ./lm/common/joint_order.hh:61 in void lm::JointOrder(const > util::stream::ChainPositions&, Callback&) [with Callback = > > lm::builder::{anonymous}::Callback<lm::builder::{anonymous}::OutputProbBackoff>; > Compare = lm::SuffixOrder] threw FormatLoadException because `order != > current + 1'. > Detected n-gram without matching suffix > -- Dingyuan Wang _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
