Your training corpus is empty. cat ~/corpus/news-commentary-v8.fr-en.true.en
On 4/22/21 9:50 PM, Namrata Hadimani wrote: > Hi All, > > I am trying to perform Language Model Training using the below command > > ~/mosesdecoder/bin/lmplz -o 3 <~/corpus/news-commentary-v8.fr-en.true.en > > news-commentary-v8.fr-en.arpa.en > > > followed similar to the link below : > http://www.statmt.org/moses/?n=Moses.Baseline > <http://www.statmt.org/moses/?n=Moses.Baseline> > > But I faced this issue : > === 1/5 Counting and sorting n-grams === > Reading /home/namrata/smt/corpus/news-commentary-v8.fr-en.true.en > ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 > tcmalloc: large alloc 3135389696 bytes == 0x144e000 @ > tcmalloc: large alloc 10451279872 bytes == 0xbc272000 @ > Unigram tokens 0 types 3 > === 2/5 Calculating and sorting adjusted counts === > Chain sizes: 1:36 2:4734547456 3:8877277184 > tcmalloc: large alloc 8877277184 bytes == 0x144e000 @ > tcmalloc: large alloc 4734550016 bytes == 0x32ba4e000 @ > terminate called after throwing an instance of > 'lm::builder::BadDiscountException' > what(): /home/namrata/smt/mosesdecoder/lm/builder/adjust_counts.cc:53 > in void > lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const > lm::builder::DiscountConfig&) threw BadDiscountException because `s.n[j] > == 0'. > Could not calculate Kneser-Ney discounts for 1-grams with adjusted count > 2 because we didn't observe any 1-grams with adjusted count 1; Is this > small or artificial data? > Try deduplicating the input. To override this error for e.g. a > class-based model, rerun with --discount_fallback > > So I tried changing my command to the following as > : ~/mosesdecoder/bin/lmplz -o 3 > <~/corpus/news-commentary-v8.fr-en.true.en > > news-commentary-v8.fr-en.arpa.en *--**discount_fallback* > > After this When i run the command : ~/mosesdecoder/bin/build_binary \ > > news-commentary-v8.fr-en.arpa.en \ > news-commentary-v8.fr-en.blm.en > > > > I am facing error like: lm/vocab.cc:324 in void > lm::ngram::MissingSentenceMarker(const lm::ngram::Config&, const char*) > threw SpecialWordMissingException. > The ARPA file is missing </s> and the model is configured to reject > these models. Run build_binary -s to disable this check. Byte: 66 > ERROR > > Could anyone help me out with this, please. > > Regards, > Namrata Hadimani > > On Thu, 22 Apr 2021 at 18:32, Namrata Hadimani > <namrata.hadim...@mycit.ie <mailto:namrata.hadim...@mycit.ie>> wrote: > > Hi Hieu, > > Thanks for the help, I am able to successfully compile the moses > ToolKit. > > Regards, > Namrata Hadimani > > On Thu, 22 Apr 2021 at 17:23, Hieu Hoang <hieuho...@gmail.com > <mailto:hieuho...@gmail.com>> wrote: > > i've just successfully compiled moses-4.0 on ubuntu 20.04 with > boost 1.71. > > there's something wrong with your boost installation > > On 4/22/2021 3:09 AM, ram anirudh cherukupally wrote: >> There is atleast 60 GB space, so I think it is not space >> issue. Has moses-4.0 been tested for compilation using boost >> 1.71? Do you recommend using boost 1.64 (as exemplified in >> Moses manual?) >> Thank you >> >> On Thu, Apr 22, 2021 at 1:15 PM Hieu Hoang >> <hieuho...@gmail.com <mailto:hieuho...@gmail.com>> wrote: >> >> there seems to be a problem with the boost library. Is the >> disk full? Perhaps you should re-install boost >> >> On 4/21/2021 10:39 PM, ram anirudh cherukupally wrote: >>> Dear Moses-Support, >>> >>> Please find the build.log.gz attached as per the >>> instructions when the build failed. >>> My system specs: >>> >>> OS: Ubuntu 20.04 >>> RAM: 8 GB >>> libboost-dev version: 1.71.0.0ubuntu2 >>> >>> Command used for compiling moses: ./bjam -j4 >>> >>> Thanks and Regards >>> >>> -- >>> CH Ram Anirudh >>> >>> >>> _______________________________________________ >>> Moses-support mailing list >>> Moses-support@mit.edu <mailto:Moses-support@mit.edu> >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> <http://mailman.mit.edu/mailman/listinfo/moses-support> >> >> -- >> Hieu Hoang >> http://statmt.org/hieu <http://statmt.org/hieu> >> >> >> >> -- >> CH Ram Anirudh >> > -- > Hieu Hoang > http://statmt.org/hieu <http://statmt.org/hieu> > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu <mailto:Moses-support@mit.edu> > http://mailman.mit.edu/mailman/listinfo/moses-support > <http://mailman.mit.edu/mailman/listinfo/moses-support> > > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
