Hi, oh, that explains.
The pipe character "|" is used in Moses to separate factors. Categorically, it is very bad to have pipe characters in training, tuning, testing, and language model data, and causes problems all over the place. If you really need it, I suggest to replace it by &bar; during tokenization. -phi On Mon, Mar 21, 2011 at 10:45 AM, Andreas Kull <[email protected]> wrote: > Hi, > > I check all lines and verified that this only occurs on lines which > contain a pipe "|", e.g. "Das. Ist | ein test" will produce an error. > > > Regards, > Andreas > > On Fri, Mar 18, 2011 at 7:44 PM, Philipp Koehn <[email protected]> wrote: >> Hi, >> >> can me send me a line that the truecaser is choking on? >> That would be line 33749 in your example... >> >> -phi >> >> On Wed, Mar 16, 2011 at 12:24 PM, Andreas Kull <[email protected]> >> wrote: >>> So my issue is the following: >>> >>> I got tons of error messages in the form of: >>> >>> Use of uninitialized value $word in print at >>> /home/user/mosesdecoder/scripts-20110315-0151/recaser/truecase.perl >>> line 53, <STDIN> line 33749. >>> Use of uninitialized value $otherfactors in print at >>> /home/user/mosesdecoder/scripts-20110315-0151/recaser/truecase.perl >>> line 55, <STDIN> line 33749. >>> >>> >>> This happens if I try to truecase the Europarl corpus via: >>> >>> truecase.perl --model ../eka.de.tcm < eka.train.de > eka.tc.de >>> >>> wc -l shows the same number of lines, so do I have to be concerned >>> with this messages or can I ignore them? >>> >>> >>> Regards, >>> Andreas >>> >>> On Wed, Mar 16, 2011 at 12:31 PM, Barry Haddow <[email protected]> wrote: >>>> >>>>> you said "train the language model on truecased data" which means I >>>>> have to train a truecase model first, pipe my training data to >>>>> truecase.perl. Correct? >>>>> >>>> >>>> Yes, exactly. >>>> >>>> -- >>>> The University of Edinburgh is a charitable body, registered in >>>> Scotland, with registration number SC005336. >>>> >>>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >> > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
