correction: Adding UnknownWordPenalty to my moses.ini corrected this issue and the decoder chose the expected translation. A backtrace indicated that ProcessOneUnknownWord was trying to assign a score but there was no UnknownWordPenalty feature (and hence a score).
On Tue, May 19, 2015 at 2:49 PM, kwame porter robinson < [email protected]> wrote: > Hi, > > From the toy phrase based tutorial at [1], I'm using a modified phrase > table (I replaced the German with phrases in a different source language, > the target language is still english). And I'm finding that a certain > source phrase causes the decoder to seg fault. > > Digging deeper into the verbose output I see that after the translation > option collection it never enters the hypothesis stage even though there is > at least one path available (see the verbose output below). The source > phrase has 8 tokens and the translation "this is a" covers [0-5][6-7]. > > What would prevent the decoder from entering the hypothesis generation > stage? I did notice that the last 3 lines ([1 ; 1-1], etc) are not present > in successful source translation output. Thank you for your time. > > [1] http://www.statmt.org/moses/?n=Moses.Tutorial > > cheers > kwame > --- > Defined parameters (per moses.ini or switch): > beam-threshold: 0 > config: phrase-model/moses.ini > feature: KENLM name=LM factor=0 order=3 num-features=1 > path=/home/kwame/Moses/moses-linux-64bit/sample-models/lm/europarl.srilm.gz > WordPenalty PhraseDictionaryMemory input-factor=0 output-factor=0 > path=/home/kwame/Moses/moses-linux-64bit/sample-models/phrase-model/fixed.sorted.bit > num-features=1 table-limit=100 > input-factors: 0 > mapping: T 0 > report-segmentation: > stack: 1000 > verbose: 3 2 > weight: WordPenalty0= 0 LM= 1 PhraseDictionaryMemory0= 1 > input type is: text input > line=KENLM name=LM factor=0 order=3 num-features=1 > path=/home/kwame/Moses/moses-linux-64bit/sample-models/lm/europarl.srilm.gz > FeatureFunction: LM start: 0 end: 0 > Loading the LM will be faster if you build a binary file. > Reading > /home/kwame/Moses/moses-linux-64bit/sample-models/lm/europarl.srilm.gz > > ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 > **The ARPA file is missing <unk>. Substituting log10 probability -100.000. > > ************************************************************************************************** > line=WordPenalty > FeatureFunction: WordPenalty0 start: 1 end: 1 > line=PhraseDictionaryMemory input-factor=0 output-factor=0 > path=/home/kwame/Moses/moses-linux-64bit/sample-models/phrase-model/fixed.sorted.bit > num-features=1 table-limit=100 > FeatureFunction: PhraseDictionaryMemory0 start: 2 end: 2 > DecodeFeature: > Loading LM > Loading WordPenalty0 > Loading PhraseDictionaryMemory0 > Start loading text phrase table. Moses format : [0.824] seconds > Reading > /home/kwame/Moses/moses-linux-64bit/sample-models/phrase-model/fixed.sorted.bit > > ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 > > **************************************************************************************************** > DecodeStep(): > outputFactors=FactorMask<0> > conflictFactors=FactorMask<> > newOutputFactors=FactorMask<0> > Created input-output object : [0.825] seconds > The global weight vector looks like this: core=(1.000,0.000,1.000) > Translating line 0 in thread id 140062576498432 > Translating: 0 1 1 0 1 0 0 0 > InitializeForInput( LM ) = 0.000 > InitializeForInput( WordPenalty0 ) = 0.000 > InitializeForInput( PhraseDictionaryMemory0 ) = 0.000 > Line 0: Initialize search took 0.001 seconds total > > ... > > Translation Option Collection > a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515 > [[0..0]]core=(0.000,-1.000,0.000) > the :: term=: nonterm=: c=-5.789 core=(0.000,-1.000,-0.916) c=-5.789 > [[0..2]]core=(0.000,-1.000,-0.916) > this :: term=: nonterm=: c=-8.002 core=(0.000,-1.000,-2.303) c=-8.002 > [[0..3]]core=(0.000,-1.000,-2.303) > this is :: term=: nonterm=: c=-10.291 core=(0.000,-2.000,-0.223) > c=-10.291 [[0..5]]core=(0.000,-2.000,-0.223) > or :: term=: nonterm=: c=-7.720 core=(0.000,-1.000,-2.303) c=-7.720 > [[1..2]]core=(0.000,-1.000,-2.303) > is :: term=: nonterm=: c=-4.922 core=(0.000,-1.000,0.000) c=-4.922 > [[2..3]]core=(0.000,-1.000,0.000) > it is :: term=: nonterm=: c=-8.821 core=(0.000,-2.000,-0.223) c=-8.821 > [[2..5]]core=(0.000,-2.000,-0.223) > a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515 > [[3..3]]core=(0.000,-1.000,0.000) > is :: term=: nonterm=: c=-4.922 core=(0.000,-1.000,0.000) c=-4.922 > [[4..5]]core=(0.000,-1.000,0.000) > old :: term=: nonterm=: c=-8.945 core=(0.000,-1.000,-0.223) c=-8.945 > [[4..6]]core=(0.000,-1.000,-0.223) > a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515 > [[5..5]]core=(0.000,-1.000,0.000) > 's :: term=: nonterm=: c=-6.116 core=(0.000,-1.000,0.000) c=-6.116 > [[5..6]]core=(0.000,-1.000,0.000) > an :: term=: nonterm=: c=-6.413 core=(0.000,-1.000,0.000) c=-6.413 > [[5..6]]core=(0.000,-1.000,0.000) > a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515 > [[6..6]]core=(0.000,-1.000,0.000) > 's :: term=: nonterm=: c=-6.116 core=(0.000,-1.000,0.000) c=-6.116 > [[6..7]]core=(0.000,-1.000,0.000) > an :: term=: nonterm=: c=-6.413 core=(0.000,-1.000,0.000) c=-6.413 > [[6..7]]core=(0.000,-1.000,0.000) > a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515 > [[7..7]]core=(0.000,-1.000,0.000) > > [1 ; 1-1] > > [1 ; 2-2] > > [1 ; 4-4] > --- > > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
