Thanks for your reply. I've tried the hierarchical model using larger corpura (about 2M pairs of sentences) as described in the website and CreatOnDiskPt works fine. However, moses_chart during mert-moses.perl gives an error:
moses_chart: OnDiskWrapper.cpp:48: bool OnDiskPt::OnDiskWrapper::OpenForLoad(const std::string&): Assertion `m_fileSource.is_open()' failed. when the file path is for the glue grammar. Should I also create the on-disk glue grammar when I use an on-disk phrase table? By the way, the flag TRAINING:binarize-all seems to make it possible that running the hierarchcal models using the experiment.perl. Best regards, -- Hwidong Na <[email protected]> KLE lab, POSTECH, KOREA 2010-05-25 (화), 12:02 +0100, Philipp Koehn: > Hi, > > the on-disk rule table for the syntax/hierarchical models is described > here: http://www.statmt.org/moses/?n=Moses.SyntaxTutorial > > It is currently not integrated into the relevant scripts (filter-model.perl > and > experiment.perl), but probably should. > > -phi > > On Tue, May 25, 2010 at 3:19 AM, Hwidong Na <[email protected]> wrote: > > Hi, > > > > I'm running the ems and until now it's fine. Even though I used the new > > commentary corpus in WMT10 data, the memery requirment for moses_chart > > is over 6g. Before I move to larger corpora, it seems that I should > > search for methods to fit the model to limited memory such as on-disk > > loading of phrase-based model. > > > > I mean the latest released version (moses-2010-04-26). > > > > -- > > Hwidong Na <[email protected]> > > KLE lab, POSTECH, KOREA > > > > > > 2010-05-24 (월), 13:06 +0100, Philipp Koehn: > >> Hi, > >> > >> there have been some updates over the last two weeks, but the current > >> version > >> should work - as you also indicate. Let me know, if there are any other > >> bugs > >> in the latest revision. > >> > >> What is the "stable version" you are referring to? > >> > >> -phi > >> > >> On Mon, May 24, 2010 at 9:28 AM, Hwidong Na <[email protected]> wrote: > >> > Thanks for your reply. > >> > > >> > I found the scripts at the path. The meta file (experiment.meta) of the > >> > stable version contains "$edinburgh-script-dir", however, it replaced > >> > with "$moses-scripst-dir/ems/support" in the lastest revision (3266). > >> > > >> > Best regards, > >> > -- > >> > Hwidong Na <[email protected]> > >> > KLE lab, POSTECH, KOREA > >> > > >> > > >> > 2010-05-24 (월), 15:23 +1000, Suzy Howlett: > >> >> There's a consolidate-training-data.perl in > >> >> $moses-script-dir/ems/support . Could the $moses-script-dir variable be > >> >> set incorrectly? > >> >> > >> >> Suzy > >> >> > >> >> > >> >> On 24/05/10 3:15 PM, Hwidong Na wrote: > >> >> > Dear all, > >> >> > > >> >> > I have tried to run an experiment using the Experiment Mangement > >> >> > System > >> >> > (experiment.perl) with WMT10 data as described on the following url. > >> >> > > >> >> > http://www.statmt.org/moses/?n=FactoredTraining.EMS > >> >> > > >> >> > I downloaded the stable version "moses-2010-04-26", copied two files > >> >> > "config" and "experiment.meta" on my work directory, modified pathes > >> >> > to > >> >> > fit my circumstances, and executed the following command. > >> >> > > >> >> > experiment.perl -config config -exec > >> >> > > >> >> > This gave errors like > >> >> > > >> >> > TRUECASER:train(1) crashed: error > >> >> > TRUECASER:train(1) crashed: error > >> >> > step TRUECASER:train crashed > >> >> > > >> >> > and in the log file "TRUCASER_consolidate.1.STDERR" > >> >> > > >> >> > /myworkdir/steps/1/TRUECASER_consolidate.1: > >> >> > 9: /edinburgh-script-dir/consolidate-training-data.perl: not found > >> >> > > >> >> > It seemed that I didn't have the script > >> >> > "consolidate-training-data.perl". I searched for a while but not found > >> >> > the script. Where can I fount it? (or can I skip the consolidation > >> >> > step?) > >> >> > > >> >> > Best regards, > >> >> > >> > > >> > > >> > > >> > > >> > > >> > _______________________________________________ > >> > Moses-support mailing list > >> > [email protected] > >> > http://mailman.mit.edu/mailman/listinfo/moses-support > >> > > >> > >> > >> > >> > >> > > > > > > > > > > > > > > > > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
