Hi, I'm trying to tune Moses using lattice input (not confusion network input) and I'm getting some weird errors. I'd appreciate it if someone could point out where I'm going wrong.
Here's what I'm trying to do: I can train and run a model on ordinary data. I can also reorder sentences in a preprocessing step and train and run a reordered model. Both of these work without a problem. I am now trying to use the two phrase tables from these two models together in one system, and decode lattice input where the lattice contains a total of two paths, one representing the ordinary sentence and one representing the reordered version of the same sentence. In representing the lattice as a string, I order the nodes by placing the nodes of the original sentence before the nodes of the reordered sentence. Since there is no sharing of nodes except for the source and sink, I expected this would be as good an ordering as any. The "transition probability" feature value is 0 on the edge starting the original sentence, 1 on the edge starting the reordered sentence, and 0.5 on all other edges. To get this to work, I have added an extra feature to each entry in both phrase tables (0 in the ordinary table and 1 in the reordered table), binarised both phrase tables using processPhraseTable, modified moses.ini to point to both tables and add in extra [weight-t] values, modified mert-moses-new.pl, and passed the mert script the additional flags "--inputtype 2 --no-filter-phrase-table". The modification of mert-moses-new.pl consists of adding a "[ 0.0, -1.0, 1.0 ]" to "additional_triples" for the extra phrase table feature, and copying the contents of the "if ($___INPUTTYPE == 1)" clause (lines 286-291) to the "if ($___INPUTTYPE == 2)" clause below it. This was a bit of a hack, and may be part of the problem, I don't know. I'm fairly certain those were the only changes I made from running the plain non-lattice models. When I attempt to tune the model, a number of strange (probably interrelated) things happen. For many sentences, it finds no best translation. I find this odd, since it's using exactly the same data as the plain original and reordered models, and those had no problems, so I would expect both paths of the lattice to find at least one possible translation. I assume it's getting some kind of explosion in the size of the search space. Second, on some sentences I get several error messages of the form "C: got a weird distance from X to Y of Z" (e.g. for a lattice with 12 words in each of the original and reordered paths, "C: got a weird distance from 5 to 12 of 1073741822"). It looks like this error is due to the distortion part of the model, trying to reorder phrases from the two different paths of the lattice - I don't understand why it would be trying to do this. The third oddity is that the tuning process crashed with the error "Assertion failed: (! m_sourceCompleted.Overlap(m_currSourceWordsRange)), function Hypothesis, file Hypothesis.cpp, line 101." Miscellaneous additional information: When I run the regression tests "lattice-surface" and "lattice- distortion" using moses-reg-test-data-2, all tests pass. It appears to be reading in the lattices correctly. For example, for one short sentence it outputs: "Translating: word lattice: 13 0 -- (der , -100.000, 1) (der , 0.000, 7) 1 -- (höchste , -0.693, 1) 2 -- (preis , -0.693, 1) 3 -- (betrug , -0.693, 1) 4 -- (10.573 , -0.693, 1) 5 -- (huf , -0.693, 1) 6 -- (. , -0.693, 7) 7 -- (10.573 , -0.693, 1) 8 -- (preis , -0.693, 1) 9 -- (betrug , -0.693, 1) 10 -- (höchste , -0.693, 1) 11 -- (huf , -0.693, 1) 12 -- (. , -0.693, 1)" (The reordering I'm using to test this out at the moment is very simple, just swapping two words.) I haven't updated my copy of the subversion repository in a while - I have revision 2654 - so if it's simply the case that this is a problem that's been fixed since then, please let me know. Thanks, Suzy _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
