Hi, I've been playing around with the alignment model in IBM model II from GIZA.
If have found that the individual values, of a certain j,l,m combinations in a(i|j,l,m) do not sum up to one in GIZA. Even worse, some values exceed one. Is anyone familiar why this is? I would expect that we need to normalize after each iteration. An example of giza de-en/a2.1 I j l m value 0 1 5 4 0.353848 1 1 5 4 0.353848 2 1 5 4 0.292304 3 1 5 4 0.353848 4 1 5 4 0.292304 3 1 5 4 0.353848 This is more than one obviously... Regards, Sanne -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Chris Dyer Sent: woensdag 9 juli 2008 18:40 To: [EMAIL PROTECTED] Subject: Re: [Moses-support] giza question I don't do much with model 2, so I don't know exactly how this is implemented in GIZA, but there is an option: parameters modifying the models: -------------------------------- compactadtable = 1 (1: only 3-dimensional alignment table for IBM-2 and IBM-3) which reduces the parameter space by one variable. My guess is that for most corpora sizes estimating a large (IxJxMxL) parameter space is difficult and subject to overfitting, so they cut off one of the variables. If you add -compactadtable 0 to your command line, you'll get all four variables. Chris On Wed, Jul 9, 2008 at 3:39 AM, Sanne Korzec <[EMAIL PROTECTED]> wrote: > Thanks a lot, I'm not sure what went wrong, but I've got them now... > > One last question: the table produces 4 variables. > > I am assuming this is i,j,m,l from a(i|j,m,l) and then the probability. > > But one variable, l in my interpretation is fixed at 100 instead of the real > sentence length, why is this? > > Thanks, > Sanne > > > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Chris Dyer > Sent: woensdag 9 juli 2008 1:12 > To: [EMAIL PROTECTED] > Subject: Re: [Moses-support] giza question > >> >> However, when I set model2dumpfrequency to 1, the translation table is >> outputted, but not the alignment table. > > That's surprising. There isn't a 2.a.* file that's being generated? > The relevant code is model2.cpp, line 115. I'm using the following > command and getting model 2 a-tables at each iteration: > > ../GIZA++-v2/GIZA++ -m1 3 -m2 3 -model2dumpfrequency 1 > -model1dumpfrequency 0 -m3 0 -m4 0 -hmmiterations 0 -s > ./corpus/de.vcb -t ./corpus/en.vcb -CoocurrenceFile en-de.cooc -c > corpus/en-de-int-train.snt > > > * >> >> You say the alignment table is uniformly distributed, I have reason to >> believe it might not be. Are you sure, you do not mean the translation >> table? > > I think so. Fist, if you look at the perplexities between the last > iteration of model 1 and the first iteration of model 2 (when the > alignments are still uniform, as they are in model 1), you see they're > virtually the same: > > Model1: (3) TRAIN CROSS-ENTROPY 1.94339 PERPLEXITY 3.84608 > Model2: (1) TRAIN CROSS-ENTROPY 1.91221 PERPLEXITY 3.76385 > > Second, the code that initializes the distribution looks correct (see > line 48-60 of model2.cpp) for uniform distribution. > > However, there was a bug in some code that was corrected in the SVN > version of the code a while ago that was never released, I've > recreated the package with the fixes which you can download from > http://code.google.com/p/giza-pp/ . This might be causing the > non-uniform initialization in your version. Try downloading the new > version and see if this fixes it. > >> >> As I understand it, IBM model II is different from IBM model I as it does >> not only keep track of the translation between words but also the > alignments >> between positions of words, this means it uses two distributions and > tables. >> I'm having trouble getting GIZA to output the alignment table. >> >> Thanks, >> Sanne >> >> -----Original Message----- >> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Chris Dyer >> Sent: dinsdag 8 juli 2008 21:16 >> To: [EMAIL PROTECTED] >> Subject: Re: [Moses-support] giza question >> >> On Tue, Jul 8, 2008 at 12:38 PM, Sanne Korzec <[EMAIL PROTECTED]> wrote: >>> Hi mailing, >>> >>> >>> >>> I am training a word based model with IBM Model II in GIZA. I have two >>> questions regarding this: >>> >>> >>> >>> 1) Does anybody know how the alignment distribution a(i|j,m,l) is >>> initialized in GIZA? >> It's initialized uniformly. >> >>> >>> 2) Is there a way to output this table in between iterations, just >>> like modelxdumpfrequency does for the translation distribution? >>> >> set model2dumpfrequency to 1. >>> >>> >>> Thanks, >>> >>> Sanne >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >> >> > > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
