updated packedGrammar documentation for 6.0.5 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/commit/0652f03d Tree: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/tree/0652f03d Diff: http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/diff/0652f03d Branch: refs/heads/master Commit: 0652f03da9134c8ebedf5e4062764a2f5b275fdb Parents: b403216 Author: Felix Hieber <[email protected]> Authored: Thu Oct 22 14:13:57 2015 +0200 Committer: Felix Hieber <[email protected]> Committed: Thu Oct 22 14:13:57 2015 +0200 ---------------------------------------------------------------------- 6.0/packing.md | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-joshua-site/blob/0652f03d/6.0/packing.md ---------------------------------------------------------------------- diff --git a/6.0/packing.md b/6.0/packing.md index 8189c66..8d84004 100644 --- a/6.0/packing.md +++ b/6.0/packing.md @@ -18,12 +18,12 @@ The script can be found at example usage. You can then add it to a Joshua config file, simply replacing a `tm` path to the compressed text-file format with a path to the packed grammar directory (Joshua will automatically detect that -it is packed. +it is packed, since a packed grammar is a directory). -Packing the grammar requires first sorting it, which can take quite a -bit of temporary space. +Packing the grammar requires first sorting it by the rules source side, +which can take quite a bit of temporary space. -*CAVEAT*: You may run into problems packing very large hiero +*CAVEAT*: You may run into problems packing very very large Hiero grammars. Email the support list if you do. ### Examples @@ -31,7 +31,7 @@ bit of temporary space. A Hiero grammar, using the compressed text file version: tm = hiero -owner pt -maxspan 20 -path grammar.filtered.gz - + Pack it: $JOSHUA/scripts/support/grammar-packer.pl grammar.filtered.gz grammar.packed @@ -51,3 +51,24 @@ You have a different temp file location: Update the config file line: tm = hiero -owner pt -maxspan 20 -path grammar.packed + +### Using multiple packed grammars (Joshua 6.0.5) + +Packed grammars serialize their vocabularies which prevented the use of multiple +packed grammars during decoding. With Joshua 6.0.5, it is possible to use multiple packed grammars during decoding if they have the same serialized vocabulary. +This is achieved by packing these grammars jointly using a revised packing CLI. + +To pack multiple grammars: + + $JOSHUA/scripts/support/grammar-packer.pl grammar1.filtered.gz grammar2.filtered.gz [...] grammar1.packed grammar2.packed [...] + +This will produce two packed grammars with the same vocabulary. To use them in the decoder, put this in your ```joshua.config```: + + tm = hiero -owner pt -maxspan 20 -path grammar1.packed + tm = hiero -owner pt2 -maxspan 20 -path grammar2.packed + +Note the different owners. +If you are trying to load multiple packed grammars that do not have the same +vocabulary, the decoder will throw a RuntimeException at loading time: + + Exception in thread "main" java.lang.RuntimeException: Trying to load multiple packed grammars with different vocabularies! Have you packed them jointly?
