Great question! Let me know what you find from srilm-user.
On 02/22/2015 03:39 PM, koormoosh wrote: > Thanks Kenneth. I expected that passing > > -gt1min 0 -gt2min 0 -gt3min 0 -gt4min 0 .... > > to srilm prevents that pruning you mentioned. But it doesn't. Is this > what you meant by "pruning", or there exists a separate unavoidable > pruning that srilm does? > > On 23/02/2015 4:28 AM, <[email protected] > <mailto:[email protected]>> wrote: > > Send Moses-support mailing list submissions to > [email protected] <mailto:[email protected]> > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.mit.edu/mailman/listinfo/moses-support > or, via email, send a message with subject or body 'help' to > [email protected] > <mailto:[email protected]> > > You can reach the person managing the list at > [email protected] <mailto:[email protected]> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Moses-support digest..." > > > Today's Topics: > > 1. Re: SRI/KenLM kneser ney implementation (Kenneth Heafield) > 2. Re: Fwd: Fwd: SGM files (Barry Haddow) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 22 Feb 2015 07:00:05 -0500 > From: Kenneth Heafield <[email protected] > <mailto:[email protected]>> > Subject: Re: [Moses-support] SRI/KenLM kneser ney implementation > To: [email protected] <mailto:[email protected]> > Message-ID: <[email protected] > <mailto:[email protected]>> > Content-Type: text/plain; charset=windows-1252 > > SRILM prunes singletons for trigrams and above by default. You're > likely to get better answers to SRILM-specific questions on srilm-user. > > On 02/22/2015 06:28 AM, koormoosh wrote: > > Hi, > > > > I wonder if SRI does any sort of implicit pruning or refinement? > To be more > > precise, is there any way to force SRI not to prune anything (removing > > singletons, etc). I thought that my way of calling it does what I > want (not > > pruning), but then I don't know how to explain getting different > results. > > This is how I call SRI: > > > > > > ----------------------------------------------------------------------------------------------------- > > ./ngram-count -order 3 -text training.txt -write training.ngrams > > > > ./ngram-count -order 3 -read training.ngrams -lm training.binary > > -interpolate -ukndiscount -gt1min 0 -gt2min 0 -gt3min 0 > -write-binary-lm > > > > ./ngram -order 3 -lm training.binary -ppl test.txt -debug 2 > > > > am I missing/misusing something? > > > > > > -------------------------------------------------------------------------------------------------------- > > An example to show this problem: > > (Example-1): > > Test: "13 13 13" > > Training: "13 13 13 13 17" > > perplexity *matches* SRI: "2.79327" > > > > (Example-2): > > Test: "13 13 13" > > Training "13 13 13 13 13 13 17 17 17 17 17 14 14 15 15 15 16 16 16 16" > > perplexity *doesn't match* SRI: "4.51546" and what SRI returns us > "4.242". > > > > ------------------------------------------------------------------------------------------------------- > > > > Thanks in advance, > > Koorm > > > > > > > > _______________________________________________ > > Moses-support mailing list > > [email protected] <mailto:[email protected]> > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > ------------------------------ > > Message: 2 > Date: Sun, 22 Feb 2015 13:03:33 +0000 > From: Barry Haddow <[email protected] > <mailto:[email protected]>> > Subject: Re: [Moses-support] Fwd: Fwd: SGM files > To: Ricardo Cabello S?nchez > <[email protected] > <mailto:[email protected]>> > Cc: [email protected] <mailto:[email protected]> > Message-ID: <[email protected] > <mailto:[email protected]>> > Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; > format="flowed" > > Hi Ricardo > > The only documentation for EMS is on the Moses website, see > http://www.statmt.org/moses/?n=FactoredTraining.EMS plus the examples > in the Moses distribution and the one for the baseline system > (http://www.statmt.org/moses/uploads/Moses/config). The file > experiment.meta shows the templates that EMS uses to execute an > experiment, but is quite hard to read if you are new to Moses. > > For evaluating with text files rather than sgm, look at the EVALUATION > section of the baseline system config file. This should take care of > the errors you reported below, > > cheers - Barry > > Quoting Ricardo Cabello S?nchez > <[email protected] > <mailto:[email protected]>> on Sun, 22 Feb 2015 > 12:52:34 > +0100: > > > Hi, > > > > I get problems also at > > stepEVALUATION:test:reference-from-sgm crashed > > step EVALUATION:test:analysis-coverage crashed > > > > Coudl you please advice? Please find attached log files. > > > > Thanks > > > > Ricardo > > > > 2015-02-22 12:25 GMT+01:00 Ricardo Cabello S?nchez < > > [email protected] > <mailto:[email protected]>>: > > > >> Hi Barry, > >> > >> Thank you very much for your quick response. It helped a lot. Now > I can > >> train my system much better. I do not have much expertise in > scripts and > >> config files but I am doing my best. Do you know where I can find > a quick > >> guidance to know how to configure config file? > >> > >> Experiment crashed at test filter step. Could you please advice? > Please > >> find attached log file. > >> > >> Thank you and best regards, > >> > >> Ricardo > >> > >> 2015-02-22 11:55 GMT+01:00 Barry Haddow > <[email protected] <mailto:[email protected]>>: > >> > >>> Hi Ricardo > >>> > >>> You only need sgm files if you want to evaluate with the nist bleu > >>> script. If you want to use bleu to evaluate with txt files, just use > >>> multi-bleu.perl (in scripts/generic) > >>> > >>> cheers - Barry > >>> > >>> > >>> Quoting Ricardo Cabello S?nchez > <[email protected] > <mailto:[email protected]>> > >>> on Sun, 22 Feb 2015 11:44:51 +0100: > >>> > >>> Hi all, > >>>> > >>>> I do not know if my message reached you. > >>>> > >>>> Thank you > >>>> > >>>> Ricardo > >>>> > >>>> ---------- Forwarded message ---------- > >>>> From: Ricardo Cabello S?nchez > <[email protected] > <mailto:[email protected]>> > >>>> Date: 2015-02-21 19:51 GMT+01:00 > >>>> Subject: SGM files > >>>> To: [email protected] <mailto:[email protected]> > >>>> > >>>> > >>>> Hi, > >>>> > >>>> I would like to ask you how I can get sgm files for reference. > I already > >>>> have txt files in ENG and in PTG correctly aligned. Is there > any way to > >>>> run > >>>> moses without these sgm files? > >>>> > >>>> Thank you and best regards, > >>>> > >>>> Ricardo > >>>> > >>>> > >>> > >>> > >>> -- > >>> The University of Edinburgh is a charitable body, registered in > >>> Scotland, with registration number SC005336. > >>> > >>> > >>> > >> > > > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > > ------------------------------ > > _______________________________________________ > Moses-support mailing list > [email protected] <mailto:[email protected]> > http://mailman.mit.edu/mailman/listinfo/moses-support > > > End of Moses-support Digest, Vol 100, Issue 73 > ********************************************** > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
