How embarrassing. Can you try on head from github.com/kpu/kenlm ? If that fails, I can take this off list.
Kenneth On March 29, 2017 3:39:20 PM GMT+01:00, Dingyuan Wang <[email protected]> wrote: >Dear list, > >lmplz crashed on my machine recently. Command is > >lmplz -o 4 -S 70% --text zhc-simp.txt --arpa zhc.lm --prune 0 1 1 2 > >=== 1/5 Counting and sorting n-grams === >Reading /home/gumble/docs/E/corpus/zhs/zhc-simp.txt >----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 >tcmalloc: large alloc 2340552704 bytes == 0x55e7ed4f4000 @ >tcmalloc: large alloc 9362194432 bytes == 0x55e878d14000 @ >**************************************************************************************************** >Unigram tokens 886453003 types 66249 >=== 2/5 Calculating and sorting adjusted counts === >Chain sizes: 1:794988 2:1961835648 3:3678441728 4:5885507072 >tcmalloc: large alloc 5885509632 bytes == 0x55e7ed4f4000 @ >tcmalloc: large alloc 1961836544 bytes == 0x55e94c29c000 @ >tcmalloc: large alloc 3678445568 bytes == 0x55e9c1190000 @ >Statistics: >1 66249 D1=0.549028 D2=1.18255 D3+=0.99644 >2 14266408/22790840 D1=0.615082 D2=1.06095 D3+=1.47555 >3 87810872/205978808 D1=0.742285 D2=1.17282 D3+=1.49899 >4 62909089/415283792 D1=0.698985 D2=1.20588 D3+=1.54463 >Memory estimate for binary LM: >type MB >probing 3417 assuming -p 1.5 >probing 4002 assuming -r models -p 1.5 >trie 1653 without quantization >trie 908 assuming -q 8 -b 8 quantization >trie 1418 assuming -a 22 array pointer compression >trie 674 assuming -a 22 -q 8 -b 8 array pointer compression and >quantization >=== 3/5 Calculating and sorting initial probabilities === >tcmalloc: large alloc 4119576576 bytes == 0x55e94c1d8000 @ >tcmalloc: large alloc 9966813184 bytes == 0x55eaaf630000 @ >Chain sizes: 1:794988 2:228262528 3:1756217440 4:1509818136 >----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 >##**********###############################################################-----##**********++#############################################################-----##************#############################################################-----##************####################################################################************####################################################################************+###################################################################*************###################################################################*************##################################################################################### >=== 4/5 Calculating and writing order-interpolated probabilities === >Chain sizes: 1:794988 2:228262528 3:1756217440 4:1509818136 >----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 >---------------------------------------------------------------------------------------------------terminate >called after throwing an instance of 'lm::FormatLoadException' > what(): ./lm/common/joint_order.hh:61 in void lm::JointOrder(const >util::stream::ChainPositions&, Callback&) [with Callback = >lm::builder::{anonymous}::Callback<lm::builder::{anonymous}::OutputProbBackoff>; >Compare = lm::SuffixOrder] threw FormatLoadException because `order != >current + 1'. >Detected n-gram without matching suffix > > >-- >Dingyuan Wang >_______________________________________________ >Moses-support mailing list >[email protected] >http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
