Hey Tom the branch mt3_chart re-implemented a binarized phrase table using berkeley db, and redid the plumbing in the decoder to allow different phrase table implementation. It therefore works a little bit like the language model framework; which implementation to be used is specified in the ini file.
this is a alternative to building on the existing binary phrase table 2009/9/15 Tom Hoar <[email protected]> > I'm considering a new Moses branch that would allow users to create > proprietary binarized tables that only work in their deployments. The > purpose is to protect data models from use with unauthorized systems. The > project: > > 1. after creating binarized models with the processPhraseTable and > processLexicalTable steps, would encode the binarized data tables to > a proprietary format > 2. would modify the Moses PhraseDictionaryTreeAdaptor > (PhraseDictionaryTree & other as required) to a) detect open or proprietary > format b) if open format, execute to existing code, if proprietary execute > branched code to decode proprietary format for every transaction and lookup > 3. publish static mods to Moses as open source. Static mods could, > however, could point to dynamic linked libraries that could be proprietary. > > > Questions: > A) Does a branch like this exist or has anyone considered it? > B) What problems, if any, would this create in the smart pointer's ability > to search and find what it's looking for? > C) Encoding the .xxxtree and .xxx data files would require decoding with > every lookup... potentially degrading performance. Instead, would it be > possible to encode/decode only the .xxxvoc and .idx files to achieve the > same level of protection without the performance hit of decoding the data > and tree for each lookup? > > Thanks, > Tom > > > ---------- Forwarded message ---------- From: [email protected] < > [email protected]> > Date: Fri, Feb 20, 2009 at 6:53 PM > Subject: Re: [Moses-support] zlib compression and binarized tables > To: Adam Lopez <[email protected]> > > > > Excellent. Exactly what I was looking for. > > Thank you. > Tom > > On Fri, Feb 20, 2009 at 6:42 PM, Adam Lopez <[email protected]> wrote: > Yes, though not directly. PhraseDictionaryTreeAdaptor forwards its calls > to PDAImp, which is a wrapper on PhraseDictionaryTree, which uses a > PrefixTreeF > template class. PrefixTreeF reads binary files using a smart pointer > defined in FilePtr.h. > Adam > > On Fri, Feb 20, 2009 at 11:25 AM, [email protected] <[email protected]> > wrote: > > Thanks Adam. > > > Following up, it looks like PhraseDictionaryTreeAdaptor.cpp is > responsbile > > for reading the binarized phrase tables and language model. Is this > correct? > > > > > > Tom > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
