yes, an error message rather than segfault would be good. There's version checking à la kenlm on the on-disk phrase table for chart decoding, but nowhere to store such metadata in the old binary phrase table format.
On 27/06/2011 03:28, Barry Haddow wrote: > Hi Tom, Hieu > > If there have been some changes that make old formats incompatible, it would > be good to have an error message rather than a segfault. > > Tom - would you be able to get a stack trace and/or make your model available > somewhere in order to reproduce the problem? > > best regards - Barry > > On Sunday 26 Jun 2011 13:22:53 Hieu Hoang wrote: >> i believe there's been changes to the binary phrase table (to the >> support word alignment) and kenlm (speed/bug fixes). >> >> are you binarizing the LM and pt with latest binarizer, as well as >> running with the latest decoder? >> >> On 26/06/2011 11:15, Tom Hoar wrote: >>> I updated my Moses to the newest svn rev 4036 which includes the >>> segfault fix from 4030. It didn't help. My translations still generate >>> the same segmentation fault as below. >>> >>> Tom >>> >>> On Sat, 25 Jun 2011 23:34:49 +0700, Tom Hoar >>> >>> <[email protected]> wrote: >>>> I built translation model several months ago with moses svn rev 3675 >>>> (~ Oct 2010). I'm trying to run under a newer moses build (svn 4011). >>>> I re-binarized the kenlm language model to the newer supported >>>> version (4). Running moses ended in a Segmentation fault. So, I >>>> re-binarized the phrase table (this translation model does not have a >>>> reordering table). Running still ends with a segmentation fault. The >>>> Moses output running in a terminal stdin/stdout is below. >>>> >>>> Other translation models that I've trained with the new (svn 4011) >>>> build work fine. Have there been any changes that require that I >>>> re-train/re-tune this model? >>>> >>>> Thanks, >>>> Tom >>>> >>>> user@host:~$ moses -f >>>> /usr/local/etc/domy/oct-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7-l=ils_ja_ >>>> all-T=kenlm-n=5.conf -v 4 >>>> Defined parameters (per moses.ini or switch): >>>> config: >>>> /usr/local/etc/domy/oct-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7-l=ils_ja_ >>>> all-T=kenlm-n=5.conf >>>> >>>> distortion-limit: 0 >>>> input-factors: 0 >>>> lmodel-file: 9 0 5 >>>> /opt/ils/ENGINES/lms/lm-t=ja-l=ils_ja_all-T=kenlm-n=5/kenlm.ja.mm >>>> mapping: 0 T 0 >>>> threads: 4 >>>> ttable-file: 1 0 0 5 >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/ph >>>> rase-table ttable-limit: 20 >>>> v: 0 >>>> verbose: 4 >>>> weight-d: 1.71428e-06 >>>> weight-l: 0.198294 >>>> weight-t: 0.224902 0.00343444 0.1214 6.07841e-07 -0.276618 >>>> weight-w: -0.175349 >>>> xml-input: exclusive >>>> Added ScoreProducer(0 Distortion) index=0-0 >>>> input type is: text input >>>> Added ScoreProducer(1 WordPenalty) index=1-1 >>>> Added ScoreProducer(2 !UnknownWordPenalty) index=2-2 >>>> Loading lexical distortion models...have 0 models >>>> Start loading LanguageModel >>>> /opt/ils/ENGINES/lms/lm-t=ja-l=ils_ja_all-T=kenlm-n=5/kenlm.ja.mm : >>>> [0.000] seconds >>>> Added ScoreProducer(3 LM_5gram) index=3-3 >>>> Finished loading LanguageModels : [0.000] seconds >>>> Creating phrase table features >>>> Start loading PhraseTable >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/ph >>>> rase-table >>>> >>>> : [0.000] seconds >>>> >>>> filePath: >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/ph >>>> rase-table DecodeFeature: input=FactorMask<0> output=FactorMask<0> >>>> Added ScoreProducer(4 PhraseModel) index=4-8 >>>> Finished loading phrase tables : [0.000] seconds >>>> DecodeStep(): >>>> outputFactors=FactorMask<0> >>>> conflictFactors=FactorMask<> >>>> newOutputFactors=FactorMask<0> >>>> Adding decoder graph 0 to translation system default >>>> Adding language model 0 to translation system default >>>> IO from STDOUT/STDIN >>>> Created input-output object : [0.000] seconds >>>> The score component vector looks like this: >>>> Distortion >>>> WordPenalty >>>> !UnknownWordPenalty >>>> LM_5gram >>>> PhraseModel_1 >>>> PhraseModel_2 >>>> PhraseModel_3 >>>> PhraseModel_4 >>>> PhraseModel_5 >>>> >>>> The global weight vector looks like this: 0.000 -0.175 1.000 0.198 >>>> 0.225 0.003 0.121 0.000 -0.277 >>>> 種 >>>> Translating line 0 in thread id 140373814597376 >>>> Translating: 種 >>>> >>>> reading bin ttable >>>> size of OFF_T 8 >>>> binary phrasefile loaded, default OFF_T: -1 >>>> Segmentation fault >>>> user@host:~$ >>>> >>>> ま り ヰ 洋 種 >>>> >>>> user@host:~$ moses -f >>>> /usr/local/etc/domy/oct-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7-l=ils_ja_ >>>> all-T=kenlm-n=5.conf -v 4 >>>> Defined parameters (per moses.ini or switch): >>>> config: >>>> /usr/local/etc/domy/oct-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7-l=ils_ja_ >>>> all-T=kenlm-n=5.conf >>>> >>>> distortion-limit: 0 >>>> input-factors: 0 >>>> lmodel-file: 9 0 5 >>>> /opt/ils/ENGINES/lms/lm-t=ja-l=ils_ja_all-T=kenlm-n=5/kenlm.ja.mm >>>> mapping: 0 T 0 >>>> threads: 4 >>>> ttable-file: 1 0 0 5 >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/ph >>>> rase-table ttable-limit: 20 >>>> v: 0 >>>> verbose: 4 >>>> weight-d: 1.71428e-06 >>>> weight-l: 0.198294 >>>> weight-t: 0.224902 0.00343444 0.1214 6.07841e-07 -0.276618 >>>> weight-w: -0.175349 >>>> xml-input: exclusive >>>> Added ScoreProducer(0 Distortion) index=0-0 >>>> input type is: text input >>>> Added ScoreProducer(1 WordPenalty) index=1-1 >>>> Added ScoreProducer(2 !UnknownWordPenalty) index=2-2 >>>> Loading lexical distortion models...have 0 models >>>> Start loading LanguageModel >>>> /opt/ils/ENGINES/lms/lm-t=ja-l=ils_ja_all-T=kenlm-n=5/kenlm.ja.mm : >>>> [0.000] seconds >>>> Added ScoreProducer(3 LM_5gram) index=3-3 >>>> Finished loading LanguageModels : [0.000] seconds >>>> Creating phrase table features >>>> Start loading PhraseTable >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/ph >>>> rase-table >>>> >>>> : [0.000] seconds >>>> >>>> filePath: >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/ph >>>> rase-table DecodeFeature: input=FactorMask<0> output=FactorMask<0> >>>> Added ScoreProducer(4 PhraseModel) index=4-8 >>>> Finished loading phrase tables : [0.000] seconds >>>> DecodeStep(): >>>> outputFactors=FactorMask<0> >>>> conflictFactors=FactorMask<> >>>> newOutputFactors=FactorMask<0> >>>> Adding decoder graph 0 to translation system default >>>> Adding language model 0 to translation system default >>>> IO from STDOUT/STDIN >>>> Created input-output object : [0.000] seconds >>>> The score component vector looks like this: >>>> Distortion >>>> WordPenalty >>>> !UnknownWordPenalty >>>> LM_5gram >>>> PhraseModel_1 >>>> PhraseModel_2 >>>> PhraseModel_3 >>>> PhraseModel_4 >>>> PhraseModel_5 >>>> >>>> The global weight vector looks like this: 0.000 -0.175 1.000 0.198 >>>> 0.225 0.003 0.121 0.000 -0.277 >>>> 種 >>>> Translating line 0 in thread id 140373814597376 >>>> Translating: 種 >>>> >>>> reading bin ttable >>>> size of OFF_T 8 >>>> binary phrasefile loaded, default OFF_T: -1 >>>> Segmentation fault >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
