also, if you want to parallelize the scoring, I would suggest you update to today's git repos.
I've added support for writing directly to a gz file. With many parallel writes, it's likely disk IO will be the limit the speed so hopefully outputting compressed files will reduce IO. https://github.com/moses-smt/mosesdecoder/commit/049ecffcabcaf5480333251938b838bc04b8314b will update the extract and consolidate programs, and the EMS, in coming days & weeks On 09/05/2012 07:18, Marcin Junczys-Dowmunt wrote: > Hi all, > my extract.sorted and extract.inv.sorted files are around 250G each, can > I split them manually at points where the source phrase changes and > score the parts independently? Will the scores be the same as for a > single run? > > Thanks, > Marcin > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
