Matt Post created JOSHUA-315:

             Summary: Thrax keeps all rules
                 Key: JOSHUA-315
             Project: Joshua
          Issue Type: Bug
            Reporter: Matt Post
             Fix For: 6.2

When extracting rules, Thrax keeps *all* options for each target side. For 
large bitexts and common source sides (e.g., "de" for Spanish–English), there 
can be tens of thousands of translations, due to errors in the alignments and 
phenomena like garbage collection. The decoder throws out all but the top 
num_translation_options of these (default 20), but before doing so, it has to 
score all the target side options with all feature functions, include the 
language model. This slows down "warming up" of the model and means that the 
first sentences to use these items are very slow to translation.

I have updated scripts/training/ to filter out using Thrax's 
rarity penalty field, but it would be much better if Thrax were to keep only 
the most 100 frequent translation options for each source side.

This message was sent by Atlassian JIRA

Reply via email to