Hieu Hoang http://www.hoang.co.uk/hieu
On 10 December 2016 at 14:06, Shubham Khandelwal <[email protected]> wrote: > Yes, CreateOnDiskPt command executed without any error. > > There are 5 files in this phrase-table.3.folder: Misc.dat , Source.dat, > TargetColl.dat, TargetInd.dat, Vocab.dat > *Misc.dat and Vocab.dat files are empty. * > I just checked that my hard-disk memory is full as this folder took 165G > already. So may be, due to this reason those 2 files are empty. But > CreateOnDiskPt > command should throw an error of *No space left on machine *when it > stopped. > Let me know if no space on my machine, is the issue or not so that I can > go for better device having more hard-disk memory. > Good idea. Not sure who's going to do it but if you do it, please send me a patch & I'll check it in > > Also May I know that How much memory phrase-table.3.folder has in general > when CreateOnDiskPt command executes completely as phrase-table.3.gz size > is only 23GB. > I'm not too sure. Try it on a disk with 1TB and please report back what you find for future reference > > Thanking You. > > > On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang <[email protected]> wrote: > >> strange, did the CreateOnDiskPt command execute ok, ie. with no error? >> >> Does this file exist: >> /home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat >> If you do >> cat Misc.dat >> what does it say? >> >> Hieu Hoang >> http://www.hoang.co.uk/hieu >> >> On 10 December 2016 at 11:30, Shubham Khandelwal <[email protected]> >> wrote: >> >>> Hi, >>> >>> Thanks Hieu for your reply. >>> I have used *CreateOnDiskPt* to binarize the model and stored in >>> *phrase-table.3.folder *using following command: >>> >>> >>> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 phrase-table.3.gz >>> phrase-table.3.folder* >>> >>> Also I have made changes in *moses.ini.3 (*i.e. I have converted >>> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*) * >>> >>> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4 >>> path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0 >>> output-factor=0 >>> >>> Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * , >>> it gave following error after *Created input-output object*: >>> >>> *terminate called after throwing an instance of 'util::Exception'* >>> * what(): OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t >>> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception >>> because `iter == m_miscInfo.end()'.* >>> *Couldn't find value for key NumSourceFactors* >>> *Aborted (core dumped)* >>> >>> Here, I do not know that what key value should I pass and how ? Can you >>> please help me in this regard. >>> >>> Thank you so much for your help. >>> >>> Regards, >>> Shubham >>> >>> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang <[email protected]> wrote: >>> >>>> This is a hierarchical model. You must binarize with CreateOnDiskPt for >>>> this model >>>> >>>> Hieu Hoang >>>> http://www.hoang.co.uk/hieu >>>> >>>> On 9 December 2016 at 08:18, Shubham Khandelwal <[email protected]> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> Thanks. It worked now. I have created compact phrase table. >>>>> Now, when I am running it using following command: >>>>> >>>>> *~/mosesdecoder/bin/moses >>>>> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all* >>>>> >>>>> Then, after creating input-output object, it gave following >>>>> segmentation fault error: >>>>> >>>>> Created input-output object : [14.796] seconds >>>>> Ich bin ein Student >>>>> Line 0: Initialize search took 0.000 seconds total >>>>> Translating: <s> Ich bin ein Student </s> ||| [0,0]=X (1) [0,1]=X (1) >>>>> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1) >>>>> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1) >>>>> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1) >>>>> [5,5]=X (1) >>>>> >>>>> Segmentation fault (core dumped) >>>>> >>>>> In my machine, I have 40GB RAM but still I am confused why it gave >>>>> this error. >>>>> Can you please help me in this regard. I have attached moses.ini.2 for >>>>> your reference. >>>>> >>>>> Thanks. >>>>> >>>>> Regards, >>>>> Shubham >>>>> >>>>> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang <[email protected]> >>>>> wrote: >>>>> >>>>>> maybe try >>>>>> >>>>>> -encoding None >>>>>> >>>>>> On 08/12/2016 19:44, Shubham Khandelwal wrote: >>>>>> >>>>>> Hi Hieu, >>>>>> >>>>>> Thanks for your reply. >>>>>> Yes, I have used the absolute path and also I tried with -T but it >>>>>> did not work. >>>>>> Is there any other solution to this problem. >>>>>> >>>>>> Btw, Can anybody please upload the compact model of all pre-made >>>>>> models as this will take less space and also it will be very fast during >>>>>> decoding. >>>>>> >>>>>> Thanks. >>>>>> >>>>>> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> the previous email you referred to says that the directory >>>>>>> >>>>>>> * binarised-model/ * >>>>>>> >>>>>>> *must exist before you run it, otherwise it will segfault. I would >>>>>>> also use absolute path to make sure, ie. not * >>>>>>> *binarised-model/phrase-table * >>>>>>> >>>>>>> *but * >>>>>>> >>>>>>> >>>>>>> * /home/shubham/moses/binarised-model/phrase-table * >>>>>>> >>>>>>> *The previous email exchange also says you should try to add the >>>>>>> argument * >>>>>>> >>>>>>> >>>>>>> * -T . * >>>>>>> >>>>>>> Hieu Hoang >>>>>>> http://www.hoang.co.uk/hieu >>>>>>> >>>>>>> On 8 December 2016 at 15:52, Shubham Khandelwal <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> This is just the reminder of my previous email. >>>>>>>> >>>>>>>> Thanking You. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Shubham >>>>>>>> >>>>>>>> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I have just downloaded phrase-table.2.gz (18GB) de-en model >>>>>>>>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made >>>>>>>>> models. >>>>>>>>> Now, I am converting them to PhraseDictionaryCompact using >>>>>>>>> following command (for exmaple): >>>>>>>>> >>>>>>>>> >>>>>>>>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in >>>>>>>>> ~/model/phrase-table.3.gz -nscores 4 -out >>>>>>>>> binarised-model/phrase-table * >>>>>>>>> >>>>>>>>> But after passing 1/3, it gave following segementation fault error: >>>>>>>>> >>>>>>>>> *Pass 1/3: Creating hash function for rank assignment* >>>>>>>>> *Segmentation fault (core dumped)* >>>>>>>>> >>>>>>>>> I have found almost same issue on this thread: >>>>>>>>> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033 >>>>>>>>> However, I have provided the existing *binarised-model *folder in >>>>>>>>> the command. Also, I have the write-access in /tmp but still it >>>>>>>>> gave sementation fault. >>>>>>>>> >>>>>>>>> Can you please tell me what could be wrong here ? >>>>>>>>> >>>>>>>>> Thanking You. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Shubham >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Moses-support mailing list >>>>>>>> [email protected] >>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >>> >> > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
