The exact steps in Moses.Baseline are appropriate for English (and other European language) data if its starting condition is similar to the EuroParl data. Studying the data itself determines what other steps and custom work are necessary.
I've worked extensively with the Stanford Segmenter. It reliably separates Chinese characters from numbers and punctuation. Your output tokens shows some of your corpus wasn't segmented. You'll have to look at your data preparation steps to find what was missed. Other tokes indicate a closer review of your corpus is necessary and you'll probably need to build some custom cleaning scripts. Our company, PTTools is available to work with you in that study and custom script development. Tom On Wed, 8 Aug 2012 21:15:54 -0400, "Tan, Jun" <[email protected]> wrote: > Hi Tom & Barry, > > I'm most appreciative of your help. > 1) When running Moses in command line, the apostrophe (') can be > translated correctly. I think I need to check the configuration of > web > translation daemon. > 2) Just like Tom said, the corpus data is not so clean. The method is > as below: > I exactly follow the guideline > (http://www.statmt.org/moses/?n=Moses.Baseline) step by step for > English corpus preparation(tokenize, truecase, recase,clean_up), but > there is only two steps for Chinese corpus preparation (tokenize, > clean_up). The tokenize tool for Chinese is the Standford Word > Segmenter. > I don’t know what's wrong with the above method, how keep cleaning up > the corpus data. > > > -----Original Message----- > From: Tom Hoar [mailto:[email protected]] > Sent: Wednesday, August 08, 2012 4:19 PM > To: Tan, Jun > Cc: Barry Haddow; [email protected] > Subject: Re: [Moses-support] how does Moses handle with the > apostrophes? > > Thanks, Jun. > > The two files show some interesting things about the condition of > your training corpus. > > 1) You sent the source English before tokenization. So, I'm assuming > the tokenization resulted in 's for the actual Moses input. > Therefore, the presence of the 's token (ASCII apostrophe s) in your > output means > a) that token is also in the target (Chinese) corpus, and/or b) it > is not in the source corpus at all and Moses passed it through as an > unknown token. > > 2) Several output tokens are ill-formed. Some occur in the > untokenized source input, while others don't: > > HOPKINTON, => in source > Mass.— => not in source > EMC) => in source > GAAP1收益 => not in source > 现金流2958 => not in source > "我 => not in source > 。" => not in source > '' => not in source > 's => in source > reinvesting => in source > > The ones marked "in source" (including 's ) could be absent > altogether from your English training corpus. You can test for this > by running moses with the -drop-unknown option and they will > disappear from the output if they are unknown tokens. You might also > try the -unknown-lhs option to see what's in the list. > > The ones marked "not in source" indicate the output tokens are in > your target language corpus and Moses generates them from the > tables/lm during translation. Therefore, it appears your Chinese > corpus preparation (cleaning, tokenization, etc.) is not as clean as > it could be. > > Note if you normalize your source corpus with > normalize-punctuation.perl (or any other tools), you must not only > re-train your translation model, but also include it in your > translation input toolchain. > > Tom > > > On Tue, 7 Aug 2012 22:01:23 -0400, "Tan, Jun" <[email protected]> > wrote: >> Hi Barry, >> >> I have checked the source data for training. I found that some of >> the >> apostrophe already got converted to ''', but there are still some >> apostrophe like ’and [ . >> With my understanding, the tool you mentioned will convert the >> apostrophe from Unicode to ASCIII, so the tool can only works for >> the >> English-Chinese translation. Is it right? The apostrophe in >> Chinese >> is two-byte, in English is one-byte. >> If I use the tool >> (http://www.statmt.org/wmt11/normalize-punctuation.perl) , what will >> be the translation result of apostrophe(’,‘,'). >> >> >> -----Original Message----- >> From: Barry Haddow [mailto:[email protected]] >> Sent: Tuesday, August 07, 2012 6:18 PM >> To: Tan, Jun >> Cc: [email protected] >> Subject: Re: [Moses-support] how does Moses handle with the >> apostrophes? >> >> Hi Jun >> >> If you're using this version of the tokeniser on your source >> sentence, >> then I would expect it to convert the apostrophe to ' >> The fact that there is no ' in your output suggests that either the >> decoder is translating it to ' (unlikely) or the apostrophe in your >> source is not a regular apostrophe, but some unicode variant. So you >> need to check for that. >> >> This script will normalise a lot of the punctuation >> http://www.statmt.org/wmt11/normalize-punctuation.perl >> However if you use it, then you should also run it over your >> training >> data, and retrain. >> >> cheers - Barry >> >> On 07/08/12 11:00, Tan, Jun wrote: >>> Hi Barry, >>> >>> I think the version is new, below is output from the file >>> tokenizer.perl >>> #escape special chars >>> $text =~ s/\&/\&/g; # escape escape >>> $text =~ s/\|/\|/g; # factor separator >>> $text =~ s/\</\</g; # xml >>> $text =~ s/\>/\>/g; # xml >>> $text =~ s/\'/\'/g; # xml >>> $text =~ s/\"/\"/g; # xml >>> $text =~ s/\[/\[/g; # syntax non-terminal >>> $text =~ s/\]/\]/g; # syntax non-terminal >>> >>> >>> >>> -----Original Message----- >>> From: Barry Haddow [mailto:[email protected]] >>> Sent: Tuesday, August 07, 2012 5:55 PM >>> To: Tan, Jun >>> Cc: [email protected] >>> Subject: Re: [Moses-support] how does Moses handle with the >>> apostrophes? >>> >>> Hi Jun >>> >>> Recent versions of the tokeniser have a line like >>> >>> $text =~ s/\'/\'/g; # xml >>> >>> to escape apostrophes. >>> >>> cheers - Barry >>> >>> On 07/08/12 09:51, Tan, Jun wrote: >>>> Hi Barry, >>>> >>>> How to check the Moses version? I'm sure that the tokeniser for >>>> training is same with testing. I'm using Standford Word Segmenter >>>> for Chinese language. >>>> >>>> -----Original Message----- >>>> From: Barry Haddow [mailto:[email protected]] >>>> Sent: Tuesday, August 07, 2012 4:43 PM >>>> To: Tan, Jun >>>> Cc: [email protected]; [email protected] >>>> Subject: Re: [Moses-support] how does Moses handle with the >>>> apostrophes? >>>> >>>> Hi Jun >>>> >>>> Is the apostrophe in your source data an ascii apostrophe, or a >>>> unicode variant (use xxd to check this)? As Tom said, recent >>>> versions of the Moses tokeniser escape apostrophes, so either >>>> you're >>>> using an old version, or it does not recognise it as an >>>> apostrophe. >>>> >>>> Make sure you are using the same tokeniser in training and test. >>>> >>>> cheers - Barry >>>> >>>> On 07/08/12 06:38, [email protected] wrote: >>>>> Yes, I’m using Moss’s tokenizer.perl for English language, and >>>>> the >>>>> Moses got installed in June, the version should be relatively >>>>> new. >>>>> Do you have any ideas how to fix it? >>>>> From: Tom Hoar [mailto:[email protected]] >>>>> Sent: Tuesday, August 07, 2012 1:13 PM >>>>> To: Tan, Jun >>>>> Cc: [email protected] >>>>> Subject: Re: [Moses-support] how does Moses handle with the >>>>> apostrophes? >>>>> >>>>> >>>>> If you're using Moses' tokenizer.perl script, the English >>>>> handling >>>>> separates the "company's" into "company 's". In recent (~2 >>>>> months) >>>>> moses github releases, the tokenizer.perl script also escapes the >>>>> string to this "company's". The English detokenizer unescapes the >>>>> "'s" to "'s" and restores it without the preceding space. >>>>> >>>>> >>>>> >>>>> On Tue, 7 Aug 2012 00:33:07 >>>>> -0400,<[email protected]<mailto:[email protected]>> wrote: >>>>> Hi all, >>>>> >>>>> When I using Moses to translate some sentences contain >>>>> apostrophes, >>>>> it doesn’t work correctly. >>>>> Source: >>>>> EMC Corporation (NYSE:EMC) today reported strong financial >>>>> results >>>>> for the second quarter of 2012, marking the company's 10th >>>>> consecutive quarter of double-digit year-over-year growth for >>>>> consolidated revenue, GAAP net income, and GAAP and non-GAAP EPS. >>>>> EMC expects to achieve its full-year 2012 goals for consolidated >>>>> revenue, non-GAAP EPS and free cash flow. >>>>> >>>>> Translation result: >>>>> 2012 年 7 月 24 日 — EMC 公司 （ NYSE ： EMC) 今天 报告 了 强有力 的 财务 业绩 2012 年 >>>>> 第 >>>>> 2 >>>>> 季度 ， 标志 着 公司 's 连续 10 个 季度 实现 两 位 数 的 同比 增长 ， 以 实现 整合 的 收入 、 GAAP >>>>> 净 >>>>> 收入 >>>>> 和 GAAP 和 非 GAAP 每 股 收益 。 EMC 预计 到 2012 年 实现 其 目标 的 要求 年 全 年 的 合并 >>>>> 收入 >>>>> 、 >>>>> 非 GAAP EPS 和 自由 现金流 。 >>>>> >>>>> As we can see, the translation result of “company's” is “公司 >>>>> 's”，and translation of the apostrophes(‘) and the letter (s) got >>>>> failed. >>>>> Does anybody know the cause of this issue? Do I need some other >>>>> module to handle it? Does anybody know how to fix it? Below is >>>>> an >>>>> example: >>>>> >>>>> >>>>> Thanks >>>>> >>>>> >>>>> _______________________________________________ >>>>> Moses-support mailing list >>>>> [email protected] >>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> -- >>>> The University of Edinburgh is a charitable body, registered in >>>> Scotland, with registration number SC005336. >>>> >>>> >>> >>> -- >>> The University of Edinburgh is a charitable body, registered in >>> Scotland, with registration number SC005336. >>> >>> >> >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
