There’s a clean-corpus-n.perl in moses, find it and clean your corpus like:
./clean-corpus-n.perl corpus l1 l2 clean-corpus 1 100 发件人: 马洪宾 [mailto:[email protected]] 发送时间: 2012年5月16日 13:09 收件人: lixianhua 主题: Re: 答复: [Moses-support] UPDATED: moses training error I think you're right, do you have any batch to run the cleaning？ On Wed, May 16, 2012 at 12:10 PM, lixianhua <[email protected]> wrote: There must be something wrong with your extract process I suggest cleaning your corpus, as well as deleting the | [ ] characters in your corpus Then run the train script 发件人: [email protected] [mailto:[email protected]] 代表 马洪宾 发送时间: 2012年5月16日 11:28 收件人: [email protected] 主题: [Moses-support] UPDATED: moses training error Hi, I'm trying out a chinese-english baseline system using the latest moses. I'm running it on a Ubuntu server 64bit. Although I followed strictly to the tutorial http://www.statmt.org/moses/?n=Moses.Baseline, when I'm proceding the phrase " training the translation system", I get the info "ERROR: train/model/extract.o.sorted.gz does not exist in ~/working/train/model" and the program exit with exit code 2. However, I do find that there's a file named extract.sorted.gz in ~/working/train/model.(slightly different, not o.sorted.gz, but sorted.gz) $ls -l : -rw-rw-r-- 1 hongbin hongbin 30674272 May 15 16:08 aligned.grow-diag-final-and -rw-rw-r-- 1 hongbin hongbin 20 May 15 16:10 extract.inv.sorted.gz -rw-rw-r-- 1 hongbin hongbin 20 May 15 16:10 extract.sorted.gz(but the size seems to be too small) -rw-rw-r-- 1 hongbin hongbin 61246318 May 15 16:10 lex.e2f -rw-rw-r-- 1 hongbin hongbin 61246318 May 15 16:10 lex.f2e -rw-rw-r-- 1 hongbin hongbin 2 May 15 16:10 phrase-table.gz Could you please give me any clew to fix this? PS, I'm running this step by: nohup nice ~/mosesdecoder/dist/training/train-model.perl -root-dir train -corpus ~/corpus/corpus-clean -f ch -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 0:3:$HOME/lm/corpus.blm.en:8 >& training.out & (Any problem with this command?) Thanks! Hongbin -- Hongbin MA(马洪宾) Department of Computer Science and Engineering, Shanghai Jiao Tong University. Mobile: (86)188-1755-4825 _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support -- Hongbin MA(马洪宾) Department of Computer Science and Engineering, Shanghai Jiao Tong University. Mobile: (86)188-1755-4825 -- Hongbin MA(马洪宾) Department of Computer Science and Engineering, Shanghai Jiao Tong University. Mobile: (86)188-1755-4825
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
