This data set has an unusual tokenization scheme, but it works fine in the older rev.
Hieu. Yes, I re-binarized both the LM and pt with the tools shipped in svn 4011 and 4036. I still get the segfault with each new binary respectively. Yes, I can share the LM and pt. I'll compress them and send them to an FTP site of your choosing. I'll also retrain the model from original data. Thanks. Tom On Sun, 26 Jun 2011 16:37:55 -0400, Kenneth Heafield <[email protected]> wrote: > I don't change the binary file format without updating the version > number so old versions won't load. The recent versions shouldn't > impact > that. > > Sounds like a case for gdb. > > On 06/26/11 08:22, Hieu Hoang wrote: >> i believe there's been changes to the binary phrase table (to the >> support word alignment) and kenlm (speed/bug fixes). >> >> are you binarizing the LM and pt with latest binarizer, as well as >> running with the latest decoder? >> >> On 26/06/2011 11:15, Tom Hoar wrote: >>> >>> I updated my Moses to the newest svn rev 4036 which includes the >>> segfault fix from 4030. It didn't help. My translations still >>> generate >>> the same segmentation fault as below. >>> >>> >>> >>> Tom >>> >>> >>> >>> On Sat, 25 Jun 2011 23:34:49 +0700, Tom Hoar >>> <[email protected]> wrote: >>> >>>> I built translation model several months ago with moses svn rev >>>> 3675 >>>> (~ Oct 2010). I'm trying to run under a newer moses build (svn >>>> 4011). >>>> I re-binarized the kenlm language model to the newer supported >>>> version (4). Running moses ended in a Segmentation fault. So, I >>>> re-binarized the phrase table (this translation model does not >>>> have a >>>> reordering table). Running still ends with a segmentation fault. >>>> The >>>> Moses output running in a terminal stdin/stdout is below. >>>> >>>> Other translation models that I've trained with the new (svn 4011) >>>> build work fine. Have there been any changes that require that I >>>> re-train/re-tune this model? >>>> >>>> Thanks, >>>> Tom >>>> >>>> >>>> >>>> user@host:~$ moses -f >>>> >>>> /usr/local/etc/domy/oct-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7-l=ils_ja_all-T=kenlm-n=5.conf >>>> -v 4 >>>> Defined parameters (per moses.ini or switch): >>>> config: >>>> >>>> /usr/local/etc/domy/oct-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7-l=ils_ja_all-T=kenlm-n=5.conf >>>> >>>> distortion-limit: 0 >>>> input-factors: 0 >>>> lmodel-file: 9 0 5 >>>> /opt/ils/ENGINES/lms/lm-t=ja-l=ils_ja_all-T=kenlm-n=5/kenlm.ja.mm >>>> mapping: 0 T 0 >>>> threads: 4 >>>> ttable-file: 1 0 0 5 >>>> >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/phrase-table >>>> ttable-limit: 20 >>>> v: 0 >>>> verbose: 4 >>>> weight-d: 1.71428e-06 >>>> weight-l: 0.198294 >>>> weight-t: 0.224902 0.00343444 0.1214 6.07841e-07 -0.276618 >>>> weight-w: -0.175349 >>>> xml-input: exclusive >>>> Added ScoreProducer(0 Distortion) index=0-0 >>>> input type is: text input >>>> Added ScoreProducer(1 WordPenalty) index=1-1 >>>> Added ScoreProducer(2 !UnknownWordPenalty) index=2-2 >>>> Loading lexical distortion models...have 0 models >>>> Start loading LanguageModel >>>> /opt/ils/ENGINES/lms/lm-t=ja-l=ils_ja_all-T=kenlm-n=5/kenlm.ja.mm >>>> : >>>> [0.000] seconds >>>> Added ScoreProducer(3 LM_5gram) index=3-3 >>>> Finished loading LanguageModels : [0.000] seconds >>>> Creating phrase table features >>>> Start loading PhraseTable >>>> >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/phrase-table >>>> : [0.000] seconds >>>> filePath: >>>> >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/phrase-table >>>> DecodeFeature: input=FactorMask<0> output=FactorMask<0> >>>> Added ScoreProducer(4 PhraseModel) index=4-8 >>>> Finished loading phrase tables : [0.000] seconds >>>> DecodeStep(): >>>> outputFactors=FactorMask<0> >>>> conflictFactors=FactorMask<> >>>> newOutputFactors=FactorMask<0> >>>> Adding decoder graph 0 to translation system default >>>> Adding language model 0 to translation system default >>>> IO from STDOUT/STDIN >>>> Created input-output object : [0.000] seconds >>>> The score component vector looks like this: >>>> Distortion >>>> WordPenalty >>>> !UnknownWordPenalty >>>> LM_5gram >>>> PhraseModel_1 >>>> PhraseModel_2 >>>> PhraseModel_3 >>>> PhraseModel_4 >>>> PhraseModel_5 >>>> >>>> The global weight vector looks like this: 0.000 -0.175 1.000 0.198 >>>> 0.225 0.003 0.121 0.000 -0.277 >>>> 種 >>>> Translating line 0 in thread id 140373814597376 >>>> Translating: 種 >>>> >>>> reading bin ttable >>>> size of OFF_T 8 >>>> binary phrasefile loaded, default OFF_T: -1 >>>> Segmentation fault >>>> user@host:~$ >>>> >>>> ま り ヰ 洋 種 >>>> >>>> user@host:~$ moses -f >>>> >>>> /usr/local/etc/domy/oct-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7-l=ils_ja_all-T=kenlm-n=5.conf >>>> -v 4 >>>> Defined parameters (per moses.ini or switch): >>>> config: >>>> >>>> /usr/local/etc/domy/oct-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7-l=ils_ja_all-T=kenlm-n=5.conf >>>> >>>> distortion-limit: 0 >>>> input-factors: 0 >>>> lmodel-file: 9 0 5 >>>> /opt/ils/ENGINES/lms/lm-t=ja-l=ils_ja_all-T=kenlm-n=5/kenlm.ja.mm >>>> mapping: 0 T 0 >>>> threads: 4 >>>> ttable-file: 1 0 0 5 >>>> >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/phrase-table >>>> ttable-limit: 20 >>>> v: 0 >>>> verbose: 4 >>>> weight-d: 1.71428e-06 >>>> weight-l: 0.198294 >>>> weight-t: 0.224902 0.00343444 0.1214 6.07841e-07 -0.276618 >>>> weight-w: -0.175349 >>>> xml-input: exclusive >>>> Added ScoreProducer(0 Distortion) index=0-0 >>>> input type is: text input >>>> Added ScoreProducer(1 WordPenalty) index=1-1 >>>> Added ScoreProducer(2 !UnknownWordPenalty) index=2-2 >>>> Loading lexical distortion models...have 0 models >>>> Start loading LanguageModel >>>> /opt/ils/ENGINES/lms/lm-t=ja-l=ils_ja_all-T=kenlm-n=5/kenlm.ja.mm >>>> : >>>> [0.000] seconds >>>> Added ScoreProducer(3 LM_5gram) index=3-3 >>>> Finished loading LanguageModels : [0.000] seconds >>>> Creating phrase table features >>>> Start loading PhraseTable >>>> >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/phrase-table >>>> : [0.000] seconds >>>> filePath: >>>> >>>> /opt/ils/ENGINES/tables/tables-s=ja_fr10-t=ja-p=ils_ja_all-a=giza-g=7/phrase-table >>>> DecodeFeature: input=FactorMask<0> output=FactorMask<0> >>>> Added ScoreProducer(4 PhraseModel) index=4-8 >>>> Finished loading phrase tables : [0.000] seconds >>>> DecodeStep(): >>>> outputFactors=FactorMask<0> >>>> conflictFactors=FactorMask<> >>>> newOutputFactors=FactorMask<0> >>>> Adding decoder graph 0 to translation system default >>>> Adding language model 0 to translation system default >>>> IO from STDOUT/STDIN >>>> Created input-output object : [0.000] seconds >>>> The score component vector looks like this: >>>> Distortion >>>> WordPenalty >>>> !UnknownWordPenalty >>>> LM_5gram >>>> PhraseModel_1 >>>> PhraseModel_2 >>>> PhraseModel_3 >>>> PhraseModel_4 >>>> PhraseModel_5 >>>> >>>> The global weight vector looks like this: 0.000 -0.175 1.000 0.198 >>>> 0.225 0.003 0.121 0.000 -0.277 >>>> 種 >>>> Translating line 0 in thread id 140373814597376 >>>> Translating: 種 >>>> >>>> reading bin ttable >>>> size of OFF_T 8 >>>> binary phrasefile loaded, default OFF_T: -1 >>>> Segmentation fault >>>> >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
