Hello again, Yes, I was using the command as described on the Moses web site at http://www.statmt.org/moses/?n=Moses.AdvancedFeatures. I have also tried piping the results from sort through uniq before piping it into processPhraseTable and encountered the same error. Perhaps I am unaware of some option to provide to sort or uniq to alleviate this problem. At what step in the code for processPhraseTable would this error be generated?
-Megan ----- Original Message ----- From: Philipp Koehn <[EMAIL PROTECTED]> Date: Monday, July 14, 2008 1:46 am Subject: Re: [Moses-support] phrase table memory issue To: "Megan Elmore ([EMAIL PROTECTED])" <[EMAIL PROTECTED]> Cc: [email protected] > Hi, > > are you sorting the phrase table? > Check the command as described on the Moses web site. > > -phi > > On Wed, Jul 9, 2008 at 8:21 PM, Megan Elmore ([EMAIL PROTECTED]) > <[EMAIL PROTECTED]> wrote: > > Hello, > > > > Thanks very much for your quick reply. I am currently trying to > generate a binary phrase table but am getting an error: > > > > ERROR: xsource phrase already inserted (B)! > > line(17): '000 - ||| 000 รข ||| (0) (1) ||| (0) (1) ||| 0.5 > 0.540651 0.25 0.178456 2.718' > > f: 2 0 2 > > > > Does this indicate a problem with my phrase table or with the > processPhraseTable process? In the event that I need to run the > training process differently - what error or warning messages, if > any, that are generated during the training process would let me > know of any errors in my phrase table? > > > > Currently, the phrase table generated during the training process > was left in a gzip'ped format as phrase-table.0-0.gz - I am not > sure if this is relevant, but maybe the odd naming (as opposed to > just "phrase-table" listed in the online documentation) sheds light > on a step of the training process that did not complete normally > for me? > > > > -Megan > > > > ----- Original Message ----- > > From: Philipp Koehn <[EMAIL PROTECTED]> > > Date: Wednesday, July 9, 2008 2:25 pm > > Subject: Re: [Moses-support] phrase table memory issue > > To: "Megan Elmore ([EMAIL PROTECTED])" <[EMAIL PROTECTED]> > > Cc: [email protected] > > > >> Hi, > >> > >> this is a sign that the phrase table is too big to load into > memory,>> there are three options: > >> (a) use the binary phrase table > >> (b) filter the phrase table for the test set you are using > >> (c) both > >> > >> See the Moses web page for details. > >> > >> -phi > >> > >> On Wed, Jul 9, 2008 at 7:17 PM, Megan Elmore > ([EMAIL PROTECTED])>> <[EMAIL PROTECTED]> wrote: > >> > Hello, > >> > > >> > I have installed Moses and run the training process using the > >> europarl corpus but am now having problems with the decoder loading > >> the phrase table. Like a previous message on this list, I am > >> getting the error > >> > > >> > terminate called after throwing an instance of 'std::bad_alloc' > >> > what(): St9bad_alloc > >> > Aborted > >> > > >> > while the decoder is trying to load the phrase table, regardless > >> of the machine I run the decoder on (I've tried four now). Is there > >> a way I can optimize how much space the phrase table uses? Or is > >> there something that could be going wrong in the training or > >> decoding processes? I am not sure where to look for the error but > >> with a little direction I could keep trying to debug it. > >> > > >> > Thanks, > >> > -Megan E. > >> > _______________________________________________ > >> > Moses-support mailing list > >> > [email protected] > >> > http://mailman.mit.edu/mailman/listinfo/moses-support > >> > > >> > > >> > > > > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
