oh, you'll find the answer here https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/escape-special-chars.perl
* Looking for MT/NLP opportunities * Hieu Hoang http://moses-smt.org/ On 17 May 2017 at 15:48, Per Starbäck <[email protected]> wrote: > > I don#t know but I would really rule out it's not RAM issues by > eyeballing 'top' while moses is > > running. > > That is ruled out. You helped me in identifying the problem. I can > reproducibly make the same command segfault or not by just adding or > removing a single line in the phrase-table file, and you said that > that line had a syntax that wasn't allowed (anymore). That mystery is > solved. (I did run with a memory checker as well. The moses process > peaked at "Maximum resident set size (kbytes): 280360" on a computer > that right then had > 2 GB Free.) > > Even *if* there was some other problem *as well*, it is a problem that > the phrase table file I got from scripts/training/mert-moses.pl has an > old syntax with [], isn't it? > > You answered that you don't know when I asked if this is a bug in > mert-moses.pl. OK, but surely you or someone else can answer my other > question, about what the escape syntax is? I ventured a guess that it > should be "\[" and "\]". Is that right? By just including the line > > Konzert ||| [would] ||| 0.2 1 0.2 0.0833333 ||| 0-0 ||| 5 5 1 ||| ||| > > in the phrase table I got a segfault as I wrote earlier. When having > "\[would\]" there instead of "[would]" there was no crash. Is that a > good change? >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
