Hi Anne Yes, there is a version of moses which supports sparse features. It is available in the moses git repository http://sourceforge.net/scm/?type=git&group_id=171520 Look for the miramerge branch.
It was sync'ed from trunk within the last few weeks, and I hope to keep it reasonably up-to-date - it may eventually get merged back. Whilst this branch is failrly experimental, it does work, and has been used in experiments. I'm currently trying to improve the sparse feature code, so that it will use a dense representation of the core features. There are actually implementations of most, if not all, of the Chiang features, although I don't know if they're all checked into the repository. cheers - Barry Quoting Anne Schuth <[email protected]> on Wed, 7 Sep 2011 09:34:19 +0200: > Hi all, > > We are in the process of reimplementing some of the 11,001 new features of > the Chiang et al. 2009 paper. We are adding a few thousand features to our > phrase table, causing it to blow up significantly. For tuning purposes we > filter the table to only include phrases used by our tuning dataset which > brings the size on disk down to about 200MB (gzipped). However, as soon as > we load this table into memory with Moses, it takes more than 60GB. This is > not really a surprise I guess since Moses will represent all our 0's as > floating points, but it is a problem since not all machines I would like to > run this on have that much memory. > This leads to my question: does Moses support some form of sparse > representation of phrase tables? Or, how is this issue generally solved, as > I am quite sure we are not the first to try this. > > Any comments, pointers to documentation are very much appreciated! > > Best, > Anne > > -- > Anne Schuth > ILPS - ISLA - FNWI > University of Amsterdam > Science Park 904, C3.230 > 1098 XH AMSTERDAM > The Netherlands > 0031 (0) 20 525 5357 > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
