Hi, it is my understanding that currently the tree-based models (hierarchical and syntax) do not allow the dropping of unknown words. We have focused on European languages, where this is not a good idea.
We will likely add this to the syntax model. -phi 2010/5/25 dongxinghua0213 <[email protected]>: > hello, > when decoding using the follow demand : > > moses1/moses-chart-cmd/src/moses_chart -config hier-dir/tuning/moses.ini > -input-file hier-dir/evaluation/input> hier-dir/evaluation/output, > > the decoding is ok ,but to filter the unknown words ,I add edthe > -drop-unknown option , it didn't work ! > > t...@top-desktop:~/programming/language_model$ > moses1/moses-chart-cmd/src/moses_chart -drop-unknown -config > hier-dir/tuning/moses.ini -input-file hier-dir/evaluation/input> > hier-dir/evaluation/output2 > Defined parameters (per moses.ini or switch): > config: hier-dir/tuning/moses.ini > cube-pruning-pop-limit: 1000 > drop-unknown: > glue-rule-type: 0 > input-factors: 0 > input-file: hier-dir/evaluation/input > inputtype: 3 > lmodel-file: 0 0 3 > /home/top/programming/language_model/hier-dir/lm/raw.uy.lm > mapping: 0 T 0 1 T 1 > max-chart-span: 20 1000 > non-terminals: X > search-algorithm: 3 > ttable-file: 2 0 0 5 > /home/top/programming/language_model/hier-dir/model/rules.bin 6 0 0 1 > /home/top/programming/language_model/hier-dir/model/glue-grammar > ttable-limit: 20 > weight-d: 1 > weight-l: 0.373002 > weight-t: 0.000746 0.172350 0.121478 0.007132 0.082449 0.123855 > weight-w: -0.118987 > Loading lexical distortion models...have 0 models > Start loading LanguageModel > /home/top/programming/language_model/hier-dir/lm/raw.uy.lm : [0.000] seconds > Finished loading LanguageModels : [3.000] seconds > Using uniform ttable-limit of 20 for all translation tables. > Start loading PhraseTable > /home/top/programming/language_model/hier-dir/model/rules.bin : [3.000] > seconds > filePath: /home/top/programming/language_model/hier-dir/model/rules.bin > Start loading PhraseTable > /home/top/programming/language_model/hier-dir/model/glue-grammar : [3.000] > seconds > filePath: /home/top/programming/language_model/hier-dir/model/glue-grammar > Start loading new format pt model : [3.000] seconds > Finished loading phrase tables : [3.000] seconds > Created input-output object : [3.000] seconds > Translating: <s> 是 党员 就 要 学 。 当 跟 你 是 的 那么 落后 </s> ||| [0,0]=X (1) [0,1]=X > (1) [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [0,6]=X (1) [0,7]=X (1) > [0,8]=X (1) [0,9]=X (1) [0,10]=X (1) [0,11]=X (1) [0,12]=X (1) [0,13]=X (1) > [0,14]=X (1) [1,1]=X (1) [1,2]=X (1) [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) > [1,6]=X (1) [1,7]=X (1) [1,8]=X (1) [1,9]=X (1) [1,10]=X (1) [1,11]=X (1) > [1,12]=X (1) [1,13]=X (1) [1,14]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1) > [2,5]=X (1) [2,6]=X (1) [2,7]=X (1) [2,8]=X (1) [2,9]=X (1) [2,10]=X (1) > [2,11]=X (1) [2,12]=X (1) [2,13]=X (1) [2,14]=X (1) [3,3]=X (1) [3,4]=X (1) > [3,5]=X (1) [3,6]=X (1) [3,7]=X (1) [3,8]=X (1) [3,9]=X (1) [3,10]=X (1) > [3,11]=X (1) [3,12]=X (1) [3,13]=X (1) [3,14]=X (1) [4,4]=X (1) [4,5]=X (1) > [4,6]=X (1) [4,7]=X (1) [4,8]=X (1) [4,9]=X (1) [4,10]=X (1) [4,11]=X (1) > [4,12]=X (1) [4,13]=X (1) [4,14]=X (1) [5,5]=X (1) [5,6]=X (1) [5,7]=X (1) > [5,8]=X (1) [5,9]=X (1) [5,10]=X (1) [5,11]=X (1) [5,12]=X (1) [5,13]=X (1) > [5,14]=X (1) [6,6]=X (1) [6,7]=X (1) [6,8]=X (1) [6,9]=X (1) [6,10]=X (1) > [6,11]=X (1) [6,12]=X (1) [6,13]=X (1) [6,14]=X (1) [7,7]=X (1) [7,8]=X (1) > [7,9]=X (1) [7,10]=X (1) [7,11]=X (1) [7,12]=X (1) [7,13]=X (1) [7,14]=X (1) > [8,8]=X (1) [8,9]=X (1) [8,10]=X (1) [8,11]=X (1) [8,12]=X (1) [8,13]=X (1) > [8,14]=X (1) [9,9]=X (1) [9,10]=X (1) [9,11]=X (1) [9,12]=X (1) [9,13]=X (1) > [9,14]=X (1) [10,10]=X (1) [10,11]=X (1) [10,12]=X (1) [10,13]=X (1) > [10,14]=X (1) [11,11]=X (1) [11,12]=X (1) [11,13]=X (1) [11,14]=X (1) > [12,12]=X (1) [12,13]=X (1) [12,14]=X (1) [13,13]=X (1) [13,14]=X (1) > [14,14]=X (1) > [0..0]=0 1 [1..1]=984.000 1 20 [2..2]=0 0 1 [3..3]=1168.000 1 20 > [4..4]=186.000 1 20 [5..5]=21.000 1 17 [6..6]=4613.000 1 20 [7..7]=12.000 1 > 12 [8..8]=284.000 1 20 [9..9]=2134.000 1 20 [10..10]=1 20 [11..11]=1348.000 > 1 20 [12..12]=92.000 1 20 [13..13]=1.000 1 1 [14..14]=0 0 1 [0..1]=0 20 > [1..2]=moses_chart: OnDiskWrapper.cpp:196: OnDiskPt::Word* > OnDiskPt::OnDiskWrapper::ConvertFromMoses(Moses::FactorDirection, const > std::vector<unsigned int, std::allocator<unsigned int> >&, const > Moses::Word&) const: Assertion `factor' failed. > > > > > ________________________________ > 网易为中小企业免费提供企业邮箱（自主域名） > > ________________________________ > 网易为中小企业免费提供企业邮箱（自主域名） > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
