Works. Thanks.
On 2015/06/01 11:52, Kenneth Heafield wrote:
> --discount_fallback
>
> On 05/31/15 20:13, Dingyuan Wang wrote:
>> Dear all,
>>
>> When using lmplz to generate a 6-gram model for POS-tag-like data (small
>> vocabulary, no real word), lmplz sometimes failed to run depending the
>> dataset.
>>
>> The version of lmplz is built from the latest code from either
>> mosesdecoder or kenlm GitHub repo.
>>
>> Command line is like
>>
>> somedir/lmplz -o 6 -S 50% --text foo.txt --arpa foo.lm
>>
>> Here is the stderr. The failing dataset is 2894562 lines, 92M.
>>
>> === 1/5 Counting and sorting n-grams ===
>> Reading /home/gumble/[somedir]/zh-cn-nw-pos1.txt
>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>> ****************************************************************************************************
>> Unigram tokens 41634632 types 114
>> === 2/5 Calculating and sorting adjusted counts ===
>> Chain sizes: 1:1368 2:241348592 3:452528608 4:724045824 5:1055900160
>> 6:1448091648
>> /home/gumble/github/kenlm/lm/builder/adjust_counts.cc:59 in void
>> lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
>> lm::builder::DiscountConfig&) threw BadDiscountException because
>> `discounts_[i].amount[j] < 0.0 || discounts_[i].amount[j] > j'.
>> ERROR: 1-gram discount out of range for adjusted count 3: -0.2
>> Aborted (core dumped)
>>
>> I used `awk 'BEGIN {srand()} !/^$/ { if (rand() <= .0001) print }'
>> zh-cn-nw-pos1.txt` to sample a few lines, which sometimes produces a
>> similar error, or successfully runs.
>>
>> The attached failed sample produces the error "ERROR: 1-gram discount
>> out of range for adjusted count 2: -1.23077".
>>
>> Thanks.
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> [email protected]
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
