Hi Eric, many thanks for your reply. That worked I think? mert-moses-run.out said:
After default: -l mem_free=0.5G -hard Using SCRIPTS_ROOTDIR: /usr/share/moses/scripts SYNC distortionchecking weight-count for ttable-file checking weight-count for lmodel-file checking weight-count for distortion-file Executing: mkdir -p europarl/tuning Executing: /usr/share/moses/scripts/training/filter-model-given-input.pl ./filtered /home/llio/MOSES/model/moses.ini /home/llio/MOSES/europarl/tuning/input filtering the phrase tables... Sat Aug 23 16:16:48 BST 2008 Executing: mkdir -p /home/llio/MOSES/europarl/tuning/filtered Considering factor 0 Considering factor 0 It took a few seconds, which surprised me because the tutorial said: 'Note that this step can take many hours, even days, to run.' But I've ended up with a filtered folder containing -rw-r--r-- 1 llio llio 1048 2008-08-23 16:16 moses.ini -rw-r--r-- 1 llio llio 204201984 2008-08-23 16:23 phrase-table.0-0.1 (the date and time are wrong on this machine). Llio On Wed, Aug 20, 2008 at 1:12 AM, Eric Nichols <[EMAIL PROTECTED]> wrote: > Greetings, > > That line in mert-moses.pl is checking to see if moses is executable > but can't find it. > Replace moses in your command with /usr/bin/moses : > > mert-moses.pl europarl/tuning/input europarl/tuning/reference /usr/bin/moses > model/moses.ini --working-dir europarl/tuning --rootdir > /usr/share/moses/scripts >&mert-moses-run.out > > Eric Nichols > > On Wed, Aug 20, 2008 at 12:40 AM, Llio Humphreys <[EMAIL PROTECTED]> wrote: >> Dear Eric/Moses Support Group, >> >> I am using Ubuntu with 3.5GB RAM and finally got >> train-factored-phrase-model.perl to run! >> I am now on the tuning part of the tutorial, and I'm still using the >> Baseline data to test out the system on my machine. >> I adapted the command for tuning from: >> >> bin/moses-scripts/scripts-YYYYMMDD-HHMM/training/mert-moses.pl >> working-dir/tuning/input working-dir/tuning/reference >> moses/moses-cmd/src/moses working-dir/model/moses.ini --working-dir >> working-dir/tuning --rootdir bin/moses-scripts/scripts-YYYYMMDD-HHMM >> >> to >> >> mert-moses.pl europarl/tuning/input europarl/tuning/reference moses >> model/moses.ini --working-dir europarl/tuning --rootdir >> /usr/share/moses/scripts >&mert-moses-run.out >> >> I get the error message: >> >> After default: -l mem_free=0.5G -hard >> Using SCRIPTS_ROOTDIR: /usr/share/moses/scripts >> Not executable: moses at /usr/bin/mert-moses.pl line 297. >> >> mert-moses.pl line 297 is empty but the previous line says: >> >> die "Not executable: $___DECODER" if ! -x $___DECODER; >> >> Grateful for your advice. >> >> Thanks, >> Llio Humphreys >> >> >> >> >> On Thu, Aug 14, 2008 at 12:52 PM, Eric Nichols <[EMAIL PROTECTED]> wrote: >>> Greetings, >>> >>> In the moses package, I install everything into /usr/share/moses and >>> symlink the scripts and moses command into /usr/bin. >>> You can see a list of installed files by running the following command: >>> >>> # dpkg -L moses >>> >>> When you call a command like ngram-count or >>> train-factored-phrase-model.perl, you do not need to specify the full >>> path; >>> the system will be able to find it. I do not know if it is strictly >>> necessary to set -scripts-root-dir, but the value >>> /usr/share/moses/scripts works fine. >>> >>> Eric Nichols >>> >>> On Thu, Aug 14, 2008 at 8:02 PM, Llio Humphreys <[EMAIL PROTECTED]> wrote: >>>> Dear Murat, Anung, Hieu, Josh, Eric, Miles, Sara, Amittai, >>>> thank you all for your help. It is very, very much appreciated. I >>>> decided to try Eric's packages, and it looks like the installation >>>> worked. I typed some of the >>>> commands in the Baseline instructions without arguments, and the >>>> program either output to the screen that I missed some arguments or >>>> gave a description of the program. Thank you Eric!!! >>>> >>>> Following the Baseline instructions >>>> (http://www.statmt.org/wmt08/baseline.html) I have now got to the >>>> following step: >>>> >>>> Use SRILM to build language model: >>>> /path-to-srilm/bin/i686/ngram-count -order 5 -interpolate -kndiscount >>>> -text working-dir/lm/europarl.lowercased -lm >>>> working-dir/lm/europarl.lm >>>> >>>> In my case, I was in folder home/llio/MOSESMTDATA. I didn't know the >>>> path to ngram-count, but it was possible to invoke it without the >>>> path: >>>> >>>> ngram-count -order 5 -interpolate -kndiscount -text >>>> europarl/lm/europarl.lowercased -lm europarl/lm/europarl.lm >>>> >>>> I'm concerned about two things: >>>> 1) this ngram-count step is taking a very long time. I think I started >>>> it off around 6pm yesterday, but it's still going. It's very >>>> resource-intensive, and it's difficult to get to other windows open. >>>> I went to check up on it around 9pm, and couldn't find that particular >>>> terminal. I thought I had closed that terminal by mistake, so I stupidly >>>> opened another one, and entered the same command. I subsequently >>>> found that the original terminal was still open, so I closed the >>>> second one. I'm not sure if issuing this command a second time on the >>>> same program and files on a different terminal would corrupt the >>>> original ngramcount step, and whether I should start it off again, or >>>> whether starting it off again would make things worse? I looked up >>>> ngram-count >>>> (http://www.speech.sri.com/projects/srilm/manpages/ngram-count.1.html) >>>> and I don't think it outputs to any file, so I guess you have to be in >>>> the same terminal to do the next step? I opened >>>> another terminal and typed 'top' to see what processes are running, >>>> and I know that ngram-count is doing something, but whether it's doing >>>> well or stuck in a loop, I can't say. What I do find strange is that >>>> the time for ngram-count is said to be 00:58:20, and it's been going >>>> for hours.. I searched this problem in previous Moses Group emails and >>>> I understand that if I run this with order 4 instead of 5 it will run >>>> quicker with very similar results? So, can I just stop what it's >>>> doing, and run this command in the same terminal with order 4? Are >>>> there any files I need to 'touch' to ensure that it doesn't leave any >>>> stone unturned? >>>> >>>> 2) how to do the next step: >>>> >>>> >>>> bin/moses-scripts/scripts-YYYYMMDD-HHMM/training/train-factored-phrase-model.perl >>>> -scripts-root-dir bin/moses-scripts/scripts-YYYYMMDD-HHMM -root-dir >>>> working-dir -corpus working-dir/corpus/europarl.lowercased -f fr -e en >>>> -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm >>>> 0:5:working-dir/lm/europarl.lm:0 >>>> >>>> I assume that like ngram-count, I can just type in >>>> train-factored-phrase-model.perl without the full path...Do I need to >>>> set the -scripts-root-dir paramater? Are all the scripts in the same >>>> place? >>>> >>>> Thank you, >>>> >>>> Llio >>>> >>>> >>>> >>>> >>>> On 8/14/08, Murat ALPEREN <[EMAIL PROTECTED]> wrote: >>>> > Dear Llio, >>>> > >>>> > You should be okay with installing moses finally if you have installed >>>> all >>>> > tha dependant packages before. I am not aware of the 'whereis' command, >>>> but >>>> > once you train your model, your moses.ini file which is created by >>>> training >>>> > script will take care of the paths. However, you should carefully supply >>>> > paths while training your model. Before training your model, you should >>>> have >>>> > two seperate corpus files which are lowercased, sentence aligned and >>>> > accordingly tokenized (there are supplementary tools for this). Once you >>>> > have your corpus in two seperate files such as corpus.en, and corpus.fr >>>> you >>>> > will run a training perl script: train-factored-phrase-model.pl with >>>> various >>>> > parameters. If you need further help with this command after installing >>>> > moses and all training scripts, send me a reply including your exact >>>> path >>>> > for your corpus files and I will try to figure out the training command >>>> for >>>> > your paths. >>>> > >>>> > Cheers >>>> > >>>> > >>>> > On 8/13/08, Llio Humphreys <[EMAIL PROTECTED]> wrote: >>>> > > Hi Murat, >>>> > > thanks for this. I've got Ubuntu 8.04 so the Hardy Heron packages are >>>> > > what I need also >>>> > > (http://cl.naist.jp/~eric-n/ubuntu-nlp/dists/hardy/all/). >>>> > > >>>> > > I think I already got the order wrong...(sign of panic maybe?) >>>> > > I clicked on mckls deb and the package installer said it was already >>>> > installed. >>>> > > I clicked on srilm deb and the package installer said it was already >>>> > > installed, so I clicked Reinstall package. >>>> > > >>>> > > I can't find anything that says the order of installation, but note >>>> > > that the workshop baseline model requires installing giza before mckls >>>> > > Do I need to uninstall mkcls (if so how? is it just a matter of >>>> > > deleting the .exc file?) or is it enough to click on Reinstall >>>> > > package? >>>> > > >>>> > > When all this is done, how do I use Moses? Many of the commands in >>>> > > the baseline model >>>> > (http://www.statmt.org/wmt08/baseline.html) require >>>> > > pathnames to the various scripts and data: is it necessary to amend >>>> > > these commands or can I just type 'whereis' command to find what I >>>> > > need? >>>> > > >>>> > > Thanks, >>>> > > Llio >>>> > > >>>> > > >>>> > > On Wed, Aug 13, 2008 at 1:48 PM, Murat ALPEREN <[EMAIL PROTECTED]> >>>> > wrote: >>>> > > > Dear Llio, >>>> > > > >>>> > > > Eric's page will probably help you, I have installed pre-compiled >>>> debian >>>> > > > based Ubuntu - Hardy Heron packages. All the necessary binaries are >>>> > included >>>> > > > in Eric's repository which will guide you for the dependancies, that >>>> > means >>>> > > > there's an order of installation which you should follow. As far as >>>> I >>>> > > > remember you should first install srilm, then mkcls, giza and >>>> finally >>>> > moses. >>>> > > > Then you will be able to train your models or run any model on your >>>> > machine >>>> > > > >>>> > > > Regards >>>> > > > >>>> > > > >>>> > > > On 8/13/08, Anung Ariwibowo <[EMAIL PROTECTED]> wrote: >>>> > > >> >>>> > > >> Hi Llio, >>>> > > >> >>>> > > >> I can compile SRILM in Linux Ubuntu without problem. Can you post >>>> the >>>> > > >> error message here, maybe we can help. >>>> > > >> >>>> > > >> Cheers, >>>> > > >> Anung >>>> > > >> >>>> > > >> On Wed, Aug 13, 2008 at 8:29 PM, Llio Humphreys <[EMAIL PROTECTED]> >>>> > > >> wrote: >>>> > > >>> >>>> > > >>> Dear Josh/Hieu, >>>> > > >>> many thanks for your replies. The default shell is bash, and >>>> updating >>>> > > >>> the .profile file worked - thanks for that tip. I look forward to >>>> > > >>> hearing more from you about the ./model/extract.0-0.o.part* >>>> problem. >>>> > > >>> My apologies for my ignorance of Unix matters: I'd like to think >>>> of >>>> > > >>> myself as a newbie rather than one who is averse to learning about >>>> > > >>> these things, and the further information you have provided has >>>> been >>>> > > >>> useful and interesting. Hieu mentioned that Anung Ariwibowo got >>>> Moses >>>> > > >>> to work when he transferred to a Linux machine. A colleague has >>>> > > >>> kindly let me borrow a Linux/Ubuntu machine, but I have already >>>> run >>>> > > >>> into problems compiling SRILM! So, I'll see if Eric Nichols's >>>> > > >>> packages will take care of that: >>>> > > >>> >>>> > http://cl.naist.jp/~eric-n/ubuntu-nlp/dists/feisty/nlp/ >>>> > > >>> Best regards, >>>> > > >>> Llio >>>> > > >>> >>>> > > >>> >>>> > > >>> >>>> > > >>> On 8/13/08, Josh Schroeder <[EMAIL PROTECTED]> wrote: >>>> > > >>> > Hi Llio, >>>> > > >>> > >>>> > > >>> > >>>> > > >>> > > you may have already received my email on the following >>>> problem >>>> > when >>>> > > >>> > > building the language model: >>>> > > >>> > > >>>> > > >>> > > Executing: cat ./model/extract.0-0.o.part* > >>>> ./model/extract.0-0.o >>>> > > >>> > > cat: ./model/extract.0-0.o.part*: No such file or directory >>>> > > >>> > > Exit code: 1 >>>> > > >>> > > >>>> > > >>> > >>>> > > >>> > That's building the phrase table, not the language model. It >>>> seems >>>> > > >>> > like >>>> > > >>> > several people on the list are having problems with this step, >>>> so >>>> > I'm >>>> > > >>> > going >>>> > > >>> > to take a look at the training process and post something to the >>>> > list >>>> > > >>> > in the >>>> > > >>> > next day or two. >>>> > > >>> > >>>> > > >>> > >>>> > > >>> > > >>>> > > >>> > > 1. You mention that Moses does not use environment variables. >>>> > > >>> > > However, in order to get SRILM to work, I found it necessary >>>> to >>>> > > >>> > > create >>>> > > >>> > > environment variables and pass these on to SRILM's make: >>>> > > >>> > > >>>> > > >>> > > make SRILM=$PWD MACHINE_TYPE=macosx >>>> > > >>> > > >>>> > > >>> > >>>> > > >>> > >>>> > >>>> PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Users/lliohumphreys/MT/MOSESSUITE/srilm:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin:/Users/lliohumphreys/MT/MOSESSUITE/srilm/bin/macosx:/sw/bin/gawk >>>> > > >>> > > >>>> > MANPATH=/Users/lliohumphreys/MT/MOSESSUITE/srilm/man >>>> > > >>> > LC_NUMERIC=C >>>> > > >>> > > >>>> > > >>> > > In addition, I was also required to type in the following >>>> command >>>> > for >>>> > > >>> > > moses-scripts: >>>> > > >>> > > >>>> > > >>> > > export >>>> > > >>> > >>>> > > >>> > >>>> > >>>> SCRIPTS_ROOTDIR=/Users/lliohumphreys/MT/MOSESSUITE/bin/moses-scripts/scripts-20080811-1801 >>>> > > >>> > > >>>> > > >>> > > >>>> > > >>> > >>>> > > >>> > Sorry, I should have been more clear. Moses itself, the decoder >>>> > that >>>> > > >>> > loads >>>> > > >>> > a trained phrase table and language model and translates text, >>>> is a >>>> > > >>> > self-contained command-line program that doesn't require >>>> environment >>>> > > >>> > variables. >>>> > > >>> > >>>> > > >>> > Your first example is compiling SRILM. This is not part of the >>>> > Moses >>>> > > >>> > toolkit: it's a toolkit of its own for language modeling and a >>>> ton >>>> > of >>>> > > >>> > other >>>> > > >>> > stuff. We use it as one of two possible integrated language >>>> models >>>> > (the >>>> > > >>> > other is IRSTLM) with Moses. >>>> > > >>> > >>>> > > >>> > Your second example is part of the training regime. Yes, there >>>> is >>>> > some >>>> > > >>> > use >>>> > > >>> > of the SCRIPTS_ROOTDIR in the >>>> > > >>> > train-factored-phrase-model.perl, but for most >>>> > training >>>> > > >>> > support scripts that come with moses there is a flag that lets >>>> you >>>> > > >>> > specify >>>> > > >>> > SCRIPTS_ROOTDIR at the command line instead of storing it as an >>>> > > >>> > environment >>>> > > >>> > variable. In train-factored-phrase-model it's >>>> "-scripts-root-dir", >>>> > > >>> > which I >>>> > > >>> > think you've actually used in one of your other emails. >>>> > > >>> > >>>> > > >>> > >>>> > > >>> > >>>> > > >>> > > If I open a new terminal and echo these variables, most of >>>> them >>>> > are >>>> > > >>> > > blank, and PATH just gives the default bin paths. >>>> > > >>> > > >>>> > > >>> > > So, how do I make them permanent? I assume that if I want to >>>> use >>>> > > >>> > > Moses again, it needs to have access to these variables? How >>>> can >>>> > I >>>> > > >>> > > ensure that I can close the terminal, go home, open a new >>>> terminal >>>> > > >>> > > the >>>> > > >>> > > next day and get Moses working again? A colleague suggested I >>>> > update >>>> > > >>> > > the .bashrc file to update each new terminal session with >>>> these >>>> > > >>> > > environment variables. However, my Mac system does not appear >>>> to >>>> > have >>>> > > >>> > > a .bashrc system as a default, and when I created one in my >>>> home >>>> > > >>> > > directory and opened a new terminal, it did not access the >>>> .bashrc >>>> > > >>> > > file. >>>> > > >>> > > >>>> > > >>> > >>>> > > >>> > Here's some info on environment variables on the Mac, found >>>> with a >>>> > > >>> > quick >>>> > > >>> > Google search: >>>> > > >>> >>>> > > http://www.macdevcenter.com/pub/a/mac/2004/02/24/bash.html >>>> > > >>> > >>>> > > >>> > I tried it with .profile, that worked fine. Are you sure >>>> you're set >>>> > to >>>> > > >>> > use >>>> > > >>> > the bash shell? Try ' echo $SHELL ' in Terminal. >>>> > > >>> > >>>> > > >>> > >>>> > > >>> > > 2. You say that you ran the decoder on your laptop just fine, >>>> but >>>> > had >>>> > > >>> > > to change a few scripts for training. I have very basic >>>> knowledge >>>> > of >>>> > > >>> > > Unix systems and installing open-source software: would it be >>>> > > >>> > > possible >>>> > > >>> > > for you to detail the changes you did to the scripts to get >>>> it to >>>> > run >>>> > > >>> > > on a Mac? Although I need this information urgently, it may >>>> also >>>> > be >>>> > > >>> > > useful for other students who are installing Moses on a Mac >>>> and >>>> > who >>>> > > >>> > > may also have basic knowledge of Unix installation procedures. >>>> > > >>> > > >>>> > > >>> > >>>> > > >>> > I'll look into this. Mac isn't really the platform of choice >>>> for >>>> > > >>> > training a >>>> > > >>> > Moses model and I do most of my work on linux. If I recall >>>> > correctly, >>>> > > >>> > an >>>> > > >>> > Intel-based Mac should be easier to get working than a PowerPC >>>> one. >>>> > The >>>> > > >>> > *decoder* does work on my Intel-based laptop, but I haven't run >>>> a >>>> > full >>>> > > >>> > training setup locally in some time -- most of the time we're >>>> > working >>>> > > >>> > with >>>> > > >>> > so much data that I use a cluster of linux machines instead of >>>> my >>>> > Mac. >>>> > > >>> > >>>> > > >>> > As a word of caution: Moses isn't an out-of-the box translation >>>> > > >>> > solution >>>> > > >>> > for end users. It's research software undergoing active >>>> development, >>>> > so >>>> > > >>> > almost every user -- on any platform -- will need to muck >>>> around in >>>> > > >>> > the >>>> > > >>> > scripts at some point, or face a compile error or runtime >>>> crash. The >>>> > > >>> > ability >>>> > > >>> > to deal with unix/linux command line tools, and debug code and >>>> > scripts >>>> > > >>> > when >>>> > > >>> > necessary, is really important. That being said, I'll see what >>>> I can >>>> > do >>>> > > >>> > about highlighting where the scripts might have problems on the >>>> Mac. >>>> > > >>> > >>>> > > >>> > >>>> > > >>> > > 3. My final question: which is embarrasingly basic...can I >>>> use the >>>> > > >>> > > one >>>> > > >>> > > installation of Moses for different corpora, or do I need to >>>> do a >>>> > > >>> > > separate installation for each one? Can I have separate >>>> > > >>> > > installations >>>> > > >>> > > of SRILM, Giza and mckls, or should they all reference the >>>> same >>>> > > >>> > > libraries? >>>> > > >>> > > >>>> > > >>> > >>>> > > >>> > All you need to do to have moses use different corpora is >>>> point it >>>> > to >>>> > > >>> > a >>>> > > >>> > different moses.ini file. Assuming you have compiled moses with >>>> > support >>>> > > >>> > for >>>> > > >>> > the language model specified in the file (IRSTLM or SRILM), it >>>> will >>>> > > >>> > translate. You should only need one copy of giza, mkcls, >>>> irst/srilm, >>>> > > >>> > and >>>> > > >>> > moses. The code stays the same, it's the data model that's >>>> > different. >>>> > > >>> > >>>> > > >>> > -Josh >>>> > > >>> > >>>> > > >>> > >>>> > > >>> > >>>> > > >>> > -- >>>> > > >>> > The University of Edinburgh is a charitable body, registered in >>>> > > >>> > Scotland, with registration number SC005336. >>>> > > >>> > >>>> > > >>> > >>>> > > >>> _______________________________________________ >>>> > > >>> Moses-support mailing list >>>> > > >>> [email protected] >>>> > > >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> > > >>> >>>> > > >> >>>> > > >> >>>> > > >> -- >>>> > > >> barliant at {gmail.com, yahoo.com} >>>> > > >> Starting July 2008, barliant at cbn.net.id is no longer active >>>> > > >> Visit my Blog at barliant dot blogspot dot com >>>> > > >> >>>> > > >> _______________________________________________ >>>> > > >> Moses-support mailing list >>>> > > >> [email protected] >>>> > > >> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> > > >> >>>> > > > >>>> > > > >>>> > > >>>> > >>>> > >>>> _______________________________________________ >>>> Moses-support mailing list >>>> [email protected] >>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> >>> >> > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
