Hi Barry, not that odd. I switched the allocators for some data structures that do not get very big anyway to use memory instead of mmapped files. So you should only see two temporary files during the whole process, one during the second pass, and both during the third pass.
Best, Marcin > Hi Marcin > > Thanks for the fast fix. I'm running it now with "-T $PWD". The only > odd thing is that when I run > "ls -l /proc/<PID>/fd", I don't see any temporary files. > > I'll let you know how things go, > > cheers - Barry > > On 22/01/13 20:20, Marcin Junczys-Dowmunt wrote: >> Hi again, >> I just pushed a fix for that. You can now specify for instance >> >> processPhraseTableMin ... -T $PWD/tmp >> >> and it should create all temporary files in the current folder. This >> uses Kenneth's FMakeTemp. BTW: it might be useful if that function could >> check whether the prefix provided is an existing directory and if yes >> add "/tmp" automatically. >> >> This also works for processLexicalReorderingTableMin. >> >> Let me know, if you can use it now. >> Best, >> Marcin >> >> W dniu 22.01.2013 18:08, Barry Haddow pisze: >>> Hi Marcin >>> >>> I'm also having a similar problem. I'm getting a a BUS error when >>> trying to compact a fairly large (20G, gzipped) phrase table. My /tmp >>> is on root, which has about 27G of free space. >>> >>> I've tried setting the TMPDIR, but processPhraseTableMon still writes >>> to /tmp (see below). >>> >>> Any suggestions? >>> >>> cheers - Barry >>> >>> [bhaddow@skaol compact]$ export TMPDIR=$PWD; nohup nice >>> /home/bhaddow/moses/dist/124c36a/bin/processPhraseTableMin -in >>> 1009-13/phrase-table.sorted.gz -out 1009-13/phrase-table -nscores 5 >>> -threads 4 &> 1009-13.log.1 & >>> [1] 1479 >>> [bhaddow@skaol compact]$ >>> [bhaddow@skaol compact]$ ls -l /proc/1479/fd >>> total 0 >>> l-wx------ 1 bhaddow users 64 2013-01-22 17:05 0 -> /dev/null >>> l-wx------ 1 bhaddow users 64 2013-01-22 17:05 1 -> >>> /mnt/skaol3/bhaddow/experiments/accept/instance-selection/compact/1009-13.log.1 >>> >>> >>> lr-x------ 1 bhaddow users 64 2013-01-22 17:05 10 -> >>> /mnt/skaol3/bhaddow/experiments/accept/instance-selection/compact/1009-13/phrase-table.sorted.gz >>> >>> >>> l-wx------ 1 bhaddow users 64 2013-01-22 17:05 2 -> >>> /mnt/skaol3/bhaddow/experiments/accept/instance-selection/compact/1009-13.log.1 >>> >>> >>> l-wx------ 1 bhaddow users 64 2013-01-22 17:05 3 -> >>> /mnt/skaol3/bhaddow/experiments/accept/instance-selection/compact/1009-13/phrase-table.minphr >>> >>> >>> lrwx------ 1 bhaddow users 64 2013-01-22 17:05 4 -> /tmp/tmpfr3cMiG >>> (deleted) >>> lrwx------ 1 bhaddow users 64 2013-01-22 17:05 5 -> /tmp/tmpfzCTl3q >>> (deleted) >>> lrwx------ 1 bhaddow users 64 2013-01-22 17:05 6 -> /tmp/tmpfz7WVNb >>> (deleted) >>> lrwx------ 1 bhaddow users 64 2013-01-22 17:05 7 -> /tmp/tmpfv2pwyW >>> (deleted) >>> lrwx------ 1 bhaddow users 64 2013-01-22 17:05 8 -> /tmp/tmpfhag7iH >>> (deleted) >>> lrwx------ 1 bhaddow users 64 2013-01-22 17:05 9 -> /tmp/tmpfhRpI3r >>> (deleted) >>> >>> >>> >>> >>> >>> On 17/01/13 11:12, Marcin Junczys-Dowmunt wrote: >>>> That might actually be it, although I would expect a segfault or a >>>> "bad >>>> alloc" in that case. I do not have the equivalent of a -T option >>>> yet and >>>> the intermediate representations of the phrase table before >>>> compression >>>> are by default built in the system's temporary folder (although >>>> invisible). So if your temporary folder is too small it will not >>>> work. A >>>> quick work-around would be to symlink your /tmp folder to a partition >>>> where you have more space. I'd say you need at least as much space in >>>> your temporary folder as the size of the uncompressed text version of >>>> the phrase table or reordering table. >>>> >>>> W dniu 17.01.2013 11:30, Jacob Dlougach pisze: >>>>> Marcin Junczys-Dowmunt<junczys@...> writes: >>>>>> And another question, do you have by any chance a non-standard >>>>>> setup for >>>>>> your temporary folder? Particularly small, located in memory etc.? >>>>>> >>>>> If by temporary directory you mean /tmp, then it is in fact rather >>>>> small, and I >>>>> had to specify -T parameter for "sort" to make it work. Talking of >>>>> truncating >>>>> phrase table, I'll do that as soon as I can. >>>>> >>>>> _______________________________________________ >>>>> Moses-support mailing list >>>>> [email protected] >>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> _______________________________________________ >>>> Moses-support mailing list >>>> [email protected] >>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>> >>> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
