The closed-form estimates for Kneser-Ney are not well-defined on toy or
class-based data. I recommend using more training data. If this is a
class-based model, pass --discount_fallback.
Kenneth
On 02/24/2015 08:04 AM, fatma elzahraa Eltaher wrote:
> Dears,
> I get the following error in LM_toy_train.65.STDERR:
> Unigram tokens 25188 types 39
> === 2/5 Calculating and sorting adjusted counts ===
> Chain sizes: 1:468 2:322921696 3:605478272 4:968765120 5:1412782592
> /home/fatma/Desktop/Folder/mosesdecoder/lm/builder/adjust_counts.cc:50
> in void
> lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
> lm::builder::DiscountConfig&) threw BadDiscountException because `s.n[j]
> == 0'.
> Could not calculate Kneser-Ney discounts for 1-grams with adjusted count
> 4 because we didn't observe any 1-grams with adjusted count 3; Is this
> small or artificial data?
> How do I fix it?
>
>
> thank you,
>
>
>
> Fatma El-Zahraa El -Taher
>
> Teaching Assistant at Computer & System department
>
> Faculty of Engineering, Azhar University
>
> Email : [email protected] <mailto:[email protected]>
> mobile: +201141600434
>
>
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
