Hi, just a comment: EMS (experiment.perl) is re-running steps for which the script/binary has changed. So if you check out an update out of SVN, it will notice that update. The timestamp is stored in *INFO files, so you could (if you knew when you updated your versions) backtrack to the corresponding SVN release.
-phi On Wed, Sep 28, 2011 at 2:09 PM, Lane Schwartz <[email protected]> wrote: > On Wed, Sep 28, 2011 at 5:55 AM, Barry Haddow > <[email protected]> wrote: >> Hi >> >> So it looks like a date-stamped release folder is not popular. It would be >> possible to have a version stamp inserted into the binaries and scripts and >> have this read by ems - would this be useful? > > Yes, it would be nice to be able to run the moses binary with some > sort of --version flag to see what revision it was compiled from. > > >> And would a 'make install' be of any use? It would mean that anyone who >> wanted >> a date-stamped release could make one with --prefix, and would help users to >> separate files needed at runtime from those needed at build time. > > Agreed. > > >> If we do have a 'make install', should the scripts be squished into the bin >> directory, or have their directory structure preserved? If we go for the >> latter, then there would be the option of running them from a source >> directory >> too. > >> On Tuesday 27 September 2011 23:53:29 Raphael Payen wrote: >>> I like your idea of a VERSION.txt file. Maybe containing also the >>> configure line. >>> >>> By the way, there is already a make install which is configurable with >>> "configure --prefix=..." ; I dont understand if you want more than >>> this. >>> >>> Raphael >>> >>> 2011/9/27 Lane Schwartz <[email protected]>: >>> > I would like to see a true make install option. I don't care what the >>> > default location is, as long as it's configurable. >>> > >>> > At the least, I would prefer to see the binaries (and preferably >>> > scripts, too) installed to a bin directory within the moses dir >>> > instead of being spread throughout the source directories. >>> > >>> > I agree that it can be useful to know what version you're running on. >>> > But I don't think time stamped directories are the way to do it. I >>> > would propose that make install creates a VERSION.txt file that lists >>> > the svn version number, or some other version number. >>> > >>> > Lane >>> > >>> > On Tue, Sep 27, 2011 at 6:07 AM, Hieu Hoang <[email protected]> wrote: >>> >> I'd go with Ondrej and avoid any release procedure and run everything >>> >> in-situ. We can debug and commit scripts easier. >>> >> >>> >> Or have a very simple release that moves all executables and scripts >>> >> into a bin directory. If the user wants to move/rename, it's their >>> >> choice >>> >> >>> >> On 27 September 2011 16:30, Barry Haddow <[email protected]> wrote: >>> >>> Hi >>> >>> >>> >>> I think it's useful to have a date-stamped release directory, so that >>> >>> you can >>> >>> work on the code and have experiments running, but keep tracking of >>> >>> which version was used in which experiment. Many people at Edinburgh >>> >>> make a copy of >>> >>> the moses binary with the svn version number as suffix on the name. >>> >>> >>> >>> How about having a 'make install, which by default installs to a >>> >>> date-stamped >>> >>> directory, with the standard bin, lib etc. pattern, with all the >>> >>> scripts under >>> >>> bin? How big a change would that be? >>> >>> >>> >>> cheers - Barry >>> >>> >>> >>> On Friday 23 September 2011 08:46:06 Ondrej Bojar wrote: >>> >>> > Hi, >>> >>> > >>> >>> > mert-moses-new.pl is outdated, isn't it? There should be only >>> >>> > mert-moses.pl these days. >>> >>> > >>> >>> > The 'scripts releasing' is of mine and dates back to JHU workshop in >>> >>> > 2006. Back then, we were hacking the scripts and running experiments >>> >>> > at the same time, so we needed some track of what version of scripts >>> >>> > was used for that particular failing experiment. >>> >>> > >>> >>> > As soon as anyone has some spare time I'd suggest: >>> >>> > >>> >>> > - delete the 'releasing' code in the Makefile >>> >>> > - make sure the main make compiles everything: all moseses, all >>> >>> > auxiliary binaries in scripts etc. >>> >>> > - I'd avoid implementing 'make install', because all the tools in >>> >>> > scripts know where their frieds are sitting in terms of relative >>> >>> > paths. The install would need to preserve the complicated structure >>> >>> > anyway so there's not much point in having the install as 'cp -r' >>> >>> > does the same job. >>> >>> > >>> >>> > Cheers, O. >>> >>> > >>> >>> > On 09/23/2011 09:20 AM, Hieu Hoang wrote: >>> >>> > > agreed, we were having an offlist moan about the same thing. The >>> >>> > > separation between decoding& training, and release date-stamp >>> >>> > > thingy is >>> >>> > > historical and quite silly. >>> >>> > > >>> >>> > > The directories& release procedure has to be rationalised. >>> >>> > > >>> >>> > > Someone will do it eventually... >>> >>> > > >>> >>> > > On 23 September 2011 14:04, Joerg >>> >>> >>> >>> Tiedemann<[email protected]>wrote: >>> >>> > >> By the way, what is the use of a date-stamped directory anyway? >>> >>> > >> I find if rather disturbing that all the binaries and scripts are >>> >>> > >> distributed all over the place. >>> >>> > >> moses-cmd/src >>> >>> > >> moses-chart-cmd/src >>> >>> > >> misc >>> >>> > >> scripts >>> >>> > >> released scripts in a date-stamped directory of your choice >>> >>> > >> ... >>> >>> > >> >>> >>> > >> J�rg >>> >>> > >> >>> >>> > >> >>> >>> > >> On Thu, Sep 22, 2011 at 4:34 PM, Barry Haddow >>> >>> > >> >>> >>> > >> <[email protected]> wrote: >>> >>> > >>> Hi Hieu et al >>> >>> > >>> >>> >>> > >>> I replied to the OP on this, but forgot to CC it to the list. >>> >>> > >>> >>> >>> > >>> There was a change in mert some time in the summer (from Prague) >>> >>> > >>> meaning >>> >>> > >> >>> >>> > >> that >>> >>> > >> >>> >>> > >>> if you run an old mert-moses.perl with a new mert binary, you get >>> >>> > >>> an >>> >>> > >> >>> >>> > >> error >>> >>> > >> >>> >>> > >>> with exit code 3. I suspect this is the problem here since the >>> >>> > >> >>> >>> > >> scripts-rootdir >>> >>> > >> >>> >>> > >>> suggests the scripts are from March. Checking mert.log would >>> >>> > >>> confirm the diagnosis. >>> >>> > >>> >>> >>> > >>> The solution is to rerun 'make release' in the scripts directory, >>> >>> > >>> and >>> >>> > >>> use >>> >>> > >> >>> >>> > >> the >>> >>> > >> >>> >>> > >>> new scripts-rootdir. >>> >>> > >>> >>> >>> > >>> As an aside, I should say that I'm not keen on our two-level make >>> >>> > >>> setup, >>> >>> > >> >>> >>> > >> and >>> >>> > >> >>> >>> > >>> would like to find something simpler. Probably a one level make, >>> >>> > >>> with a >>> >>> > >>> conventional 'make install', but with the default being to >>> >>> > >>> install in a >>> >>> > >> >>> >>> > >> date >>> >>> > >> >>> >>> > >>> stamped directory. >>> >>> > >>> >>> >>> > >>> cheers - Barry >>> >>> > >>> >>> >>> > >>> On Thursday 22 Sep 2011 14:59:26 Hieu Hoang wrote: >>> >>> > >>>> i'm afraid i don't know what the problem is. The n-best-list and >>> >>> > >>>> ini >>> >>> > >>>> files look ok. I'm running mert and moses-chart that was checked >>> >>> > >>>> out >>> >>> > >>>> on the 13th September and ran ok. >>> >>> > >>>> >>> >>> > >>>> i'm not familiar with the mert code so i can't tell you if the >>> >>> > >>>> changes >>> >>> > >>>> you made are good. However, you're welcome to post the changes >>> >>> > >>>> to the >>> >>> > >>>> mailing list and someone might be know better than i do. >>> >>> > >>>> >>> >>> > >>>> On 22/09/2011 15:34, Prasanth K wrote: >>> >>> > >>>>> Hi Hieu, >>> >>> > >>>>> >>> >>> > >>>>> I am attaching the following files: >>> >>> > >>>>> >>> >>> > >>>>> multi-threads_tune-run1.100best.out - first 1000 lines from the >>> >>> > >>>>> multi-threaded decoder >>> >>> > >>>>> single-threads_tune-run1.100best.out - the same from the >>> >>> > >>>>> single-threaded decoder >>> >>> > >>>>> >>> >>> > >>>>> multi-threads_tune-run1.moses.ini - the ini file used at the >>> >>> > >>>>> beginning of the tuning when using threads >>> >>> > >>>>> single-threads_tune-run1.moses.ini - the ini file used at the >>> >>> > >>>>> beginning of the tuning when threads were not used >>> >>> > >>>>> single-threads_tune-run2.moses.ini - the ini file used at the >>> >>> > >>>>> beginning of the second iteration when threads were not used >>> >>> > >>>>> >>> >>> > >>>>> - Prasanth >>> >>> > >>>>> >>> >>> > >>>>> On Thu, Sep 22, 2011 at 10:19 AM, Hieu >>> >>> > >>>>> Hoang<[email protected] <mailto:[email protected]>> wrote: >>> >>> > >>>>> >>> >>> > >>>>> can you send me your ini file, and a few lines of the >>> >>> > >>>>> n-best list. For accuracy, send them as attachements, not cut& >>> >>> > >>>>> paste into >>> >>> > >>>>> the email. >>> >>> > >>>>> >>> >>> > >>>>> I'll try& see what the problem is >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> On 22 September 2011 14:57, Prasanth >>> >>> > >>>>> K<[email protected] >>> >>> > >>>>> <mailto:[email protected]>> wrote: >>> >>> > >>>>> >>> >>> > >>>>> Hi all, >>> >>> > >>>>> >>> >>> > >>>>> I am facing the same error that Cyrine Nasri mentioned >>> >>> > >>>>> in this thread. >>> >>> > >>>>> >>> >>> > >>>>> I will try and give more information that what has >>> >>> > >>>>> already >>> >>> > >>>>> been mentioned. >>> >>> > >>>>> 1. I was using a single-threaded version of moses >>> >>> > >>>>> until earlier and was having no problem with the experiments >>> >>> > >>>>> using >>> >>> > >> >>> >>> > >> EMS. >>> >>> > >> >>> >>> > >>>>> 2. I recently shifted to a multi-threaded version, >>> >>> > >>>>> and tried the same experiment again with EMS. >>> >>> > >>>>> This time, the tuning process crashes after a single >>> >>> > >>>>> iteration with exactly the same error as mentioned below. >>> >>> > >>>>> 3. I have used 10 threads for decoding in the tuning >>> >>> > >>>>> process, and am using a server with 32Gb ram for running the >>> >>> > >>>>> experiments. (Might not be related, but just thought I should >>> >>> > >>>>> mention!) >>> >>> > >>>>> >>> >>> > >>>>> I am not sure if Cyrine's problem was with the >>> >>> > >>>>> multi-threaded version as well, but could some one point out as >>> >>> > >>>>> to what might be wrong in this picture ? >>> >>> > >>>>> >>> >>> > >>>>> - Prasanth >>> >>> > >>>>> >>> >>> > >>>>> On Wed, Jul 27, 2011 at 5:59 PM, Cyrine NASRI >>> >>> > >>>>> >>> >>> > >>>>> <[email protected]<mailto:[email protected]>> >>> >>> > >> >>> >>> > >> wrote: >>> >>> > >>>>> Hello, >>> >>> > >>>>> I 'm trying to launch the mert using this command: >>> >>> > >>>>> >>> >>> > >>>>> ./mert-moses-new.pl<http://mert-moses-new.pl> >>> >>> > >>>>> /users/parole/cnasri/moses_work/corpus/source2TOK >>> >>> > >>>>> /users/parole/cnasri/moses_work/corpus/ref2TOK >>> >>> > >>>>> >>> >>> > >>>>> /users/parole/cnasri/moses_work/moses/moses-cmd/src/moses >>> >>> > >> >>> >>> > >> /users/parole/cnasri/moses_work/moses-scripts/scripts-20110727-154 >>> >>> > >>3/trai n >>> >>> > >> >>> >>> > >>>>> ing/moses.ini --working-dir >>> >>> > >>>>> /users/parole/cnasri/moses_work/tuning/ >>> >>> > >>>>> --mertdir /users/parole/cnasri/moses_work/moses/mert >>> >>> > >>>>> >>> >>> > >>>>> But there is only one iteration, after it stops >>> >>> > >>>>> and >>> >>> > >> >>> >>> > >> itshow: >>> >>> > >>>>> Peeking at the beginning of nbestlist to get order >>> >>> > >>>>> of scores: run1.best100.out >>> >>> > >>>>> The decoder returns the scores in this order: d lm >>> >>> > >>>>> w tm >>> >>> > >>>>> tm >>> >>> > >> >>> >>> > >> tm >>> >>> > >> >>> >>> > >>>>> Executing: gzip -f run1.best100.out >>> >>> > >>>>> Scoring the nbestlist. >>> >>> > >>>>> Executing: >>> >>> > >>>>> >>> >>> > >>>>> /users/parole/cnasri/moses_work/moses/mert/extractor >>> >>> > >>>>> --scconfig case:true --scfile run1.scores.dat --ffile >>> >>> > >>>>> run1.features.dat -r >>> >>> > >>>>> /users/parole/cnasri/moses_work/corpus/ref2TOK -n >>> >>> > >>>>> run1.best100.out.gz> extract.out 2> extract.err >>> >>> > >>>>> Executing: \cp -f init.opt run1.init.opt >>> >>> > >>>>> Executing: >>> >>> > >>>>> /users/parole/cnasri/moses_work/moses/mert/mert -d 6 >>> >>> > >>>>> --scconfig case:true -n 20 --ffile run1.features.dat --scfile >>> >>> > >>>>> run1.scores.dat --ifile run1.init.opt 2> mert.log Exit code: 3 >>> >>> > >>>>> Failed to run mert at ./mert-moses-new.pl >>> >>> > >>>>> <http://mert-moses-new.pl> line 752. >>> >>> > >>>>> and in the mert log : >>> >>> > >>>>> >>> >>> > >>>>> Seeding random numbers with system clock >>> >>> > >>>>> >>> >>> > >>>>> :Too few minimum weights. >>> >>> > >>>>> >>> >>> > >>>>> error could not initialize start point with >>> >>> > >>>>> >>> >>> > >>>>> I have not idea how i resolve this problem. >>> >>> > >>>>> Any idea please? >>> >>> > >>>>> >>> >>> > >>>>> Thank you >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> _______________________________________________ >>> >>> > >>>>> Moses-support mailing list >>> >>> > >>>>> >>> >>> > >>>>> [email protected]<mailto:[email protected]> >>> >>> > >>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> --- J.B.S. Haldane >>> >>> > >>>>> >>> >>> > >>>>> _______________________________________________ >>> >>> > >>>>> Moses-support mailing list >>> >>> > >>>>> [email protected]<mailto:[email protected]> >>> >>> > >>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> >>> >>> > >>>>> --- J.B.S. Haldane >>> >>> > >>> >>> >>> > >>> _______________________________________________ >>> >>> > >>> Moses-support mailing list >>> >>> > >>> [email protected] >>> >>> > >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> > >> >>> >>> > >> -- >>> >>> > >> >>> >>> > >> >>> >>> > >> ****************************************************************** >>> >>> > >>****** ********** J�rg Tiedemann >>> >>> > >> [email protected] >>> >>> > >> Dep. of Linguistics and Philology >>> >>> > >> http://stp.lingfil.uu.se/~joerg/ >>> >>> > >> Uppsala University tel: +46 >>> >>> > >> (0)18 - >>> >>> > >> 471 1412 >>> >>> > >> Box 635, SE-751 26 Uppsala/SWEDEN fax: +46 (0)18 - 471 1094 >>> >>> > > >>> >>> > > _______________________________________________ >>> >>> > > Moses-support mailing list >>> >>> > > [email protected] >>> >>> > > http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >>> >>> -- >>> >>> The University of Edinburgh is a charitable body, registered in >>> >>> Scotland, with registration number SC005336. >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> Moses-support mailing list >>> >>> [email protected] >>> >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >> >>> >> _______________________________________________ >>> >> Moses-support mailing list >>> >> [email protected] >>> >> http://mailman.mit.edu/mailman/listinfo/moses-support >>> > >>> > -- >>> > When a place gets crowded enough to require ID's, social collapse is not >>> > far away. It is time to go elsewhere. The best thing about space travel >>> > is that it made it possible to go elsewhere. >>> > -- R.A. Heinlein, "Time Enough For Love" >>> > >>> > _______________________________________________ >>> > Moses-support mailing list >>> > [email protected] >>> > http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >> >> -- >> Barry Haddow >> University of Edinburgh >> +44 (0) 131 651 3173 >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > > > > -- > When a place gets crowded enough to require ID's, social collapse is not > far away. It is time to go elsewhere. The best thing about space travel > is that it made it possible to go elsewhere. > -- R.A. Heinlein, "Time Enough For Love" > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
