Here's the output based on the "long form" of the name... It appears to me that at least for this test case, pmi is the only one that is differing between the long and the short form of the name (see diff runs after results)...when you run the long form of the pmi name you should get a bunch of warnings about the exponentiation value not being set...
marimba(88): more test.pmi1 37 good<>day<>for<>1 10.4189 1 1 1 1 1 1 1 day<>for<>testing<>1 10.4189 1 1 1 1 1 1 1 that<>I<>went<>2 9.4189 1 1 2 1 1 1 1 a<>good<>day<>3 8.8339 1 3 1 1 1 1 1 think<>i<>will<>3 8.8339 1 1 3 1 1 1 1 for<>testing<>i<>3 8.8339 1 1 1 3 1 1 1 testing<>i<>think<>3 8.8339 1 1 3 1 1 1 1 with<>this<>this<>4 8.4189 1 1 2 2 1 1 1 many<>things<>with<>4 8.4189 1 2 2 1 2 1 1 of<>mine<>that<>4 8.4189 1 2 2 1 2 1 1 I<>went<>to<>4 8.4189 1 2 1 2 1 1 1 things<>I<>have<>4 8.4189 1 2 2 1 1 1 1 things<>with<>this<>4 8.4189 1 2 1 2 1 1 1 mine<>that<>I<>4 8.4189 1 2 1 2 1 1 1 i<>want<>to<>5 7.8339 1 3 1 2 1 1 1 is<>a<>good<>5 7.8339 1 2 3 1 2 1 1 I<>have<>a<>5 7.8339 1 2 1 3 1 1 1 mine<>i<>want<>5 7.8339 1 2 3 1 1 1 1 this<>this<>is<>5 7.8339 1 3 2 1 1 1 1 to<>test<>many<>6 7.4189 1 1 4 2 1 1 2 many<>things<>I<>6 7.4189 1 2 2 2 2 1 1 will<>test<>many<>6 7.4189 1 1 4 2 1 1 2 i<>think<>i<>7 7.2490 1 3 1 3 1 1 1 want<>to<>test<>8 7.0970 1 1 2 5 1 1 2 went<>to<>test<>8 7.0970 1 1 2 5 1 1 2 of<>mine<>i<>9 6.8339 1 2 2 3 2 1 1 have<>a<>test<>10 6.5120 1 1 3 5 1 1 2 i<>will<>test<>10 6.5120 1 3 1 5 1 1 1 test<>many<>things<>11 6.4189 2 4 2 2 2 2 2 test<>of<>mine<>11 6.4189 2 4 2 2 2 2 2 this<>is<>a<>12 6.2490 2 3 2 3 2 2 2 a<>test<>of<>13 5.8339 2 3 4 2 2 2 2 is<>a<>test<>14 5.5120 1 2 3 5 2 1 2 marimba(89): more test.ll1 37 test<>many<>things<>1 25.5768 2 4 2 2 2 2 2 test<>of<>mine<>1 25.5768 2 4 2 2 2 2 2 this<>is<>a<>2 23.5426 2 3 2 3 2 2 2 many<>things<>with<>3 21.9829 1 2 2 1 2 1 1 of<>mine<>that<>3 21.9829 1 2 2 1 2 1 1 a<>test<>of<>4 21.8771 2 3 4 2 2 2 2 many<>things<>I<>5 19.2675 1 2 2 2 2 1 1 good<>day<>for<>6 18.3891 1 1 1 1 1 1 1 day<>for<>testing<>6 18.3891 1 1 1 1 1 1 1 of<>mine<>i<>7 18.2798 1 2 2 3 2 1 1 is<>a<>good<>8 18.1639 1 2 3 1 2 1 1 is<>a<>test<>9 17.9821 1 2 3 5 2 1 2 to<>test<>many<>10 16.4378 1 1 4 2 1 1 2 will<>test<>many<>10 16.4378 1 1 4 2 1 1 2 that<>I<>went<>11 15.6165 1 1 2 1 1 1 1 want<>to<>test<>12 15.2528 1 1 2 5 1 1 2 went<>to<>test<>12 15.2528 1 1 2 5 1 1 2 a<>good<>day<>13 14.5700 1 3 1 1 1 1 1 think<>i<>will<>13 14.5700 1 1 3 1 1 1 1 for<>testing<>i<>13 14.5700 1 1 1 3 1 1 1 testing<>i<>think<>13 14.5700 1 1 3 1 1 1 1 with<>this<>this<>14 12.9011 1 1 2 2 1 1 1 I<>went<>to<>14 12.9011 1 2 1 2 1 1 1 things<>I<>have<>14 12.9011 1 2 2 1 1 1 1 things<>with<>this<>14 12.9011 1 2 1 2 1 1 1 mine<>that<>I<>14 12.9011 1 2 1 2 1 1 1 i<>want<>to<>15 11.9134 1 3 1 2 1 1 1 I<>have<>a<>15 11.9134 1 2 1 3 1 1 1 mine<>i<>want<>15 11.9134 1 2 3 1 1 1 1 this<>this<>is<>15 11.9134 1 3 2 1 1 1 1 have<>a<>test<>16 11.6157 1 1 3 5 1 1 2 i<>think<>i<>17 10.9864 1 3 1 3 1 1 1 i<>will<>test<>18 10.0516 1 3 1 5 1 1 1 marimba(90): more test.ps1 37 test<>many<>things<>1 8.2848 2 4 2 2 2 2 2 test<>of<>mine<>1 8.2848 2 4 2 2 2 2 2 this<>is<>a<>2 8.0492 2 3 2 3 2 2 2 a<>test<>of<>3 7.4739 2 3 4 2 2 2 2 good<>day<>for<>4 6.2218 1 1 1 1 1 1 1 day<>for<>testing<>4 6.2218 1 1 1 1 1 1 1 that<>I<>went<>5 5.5287 1 1 2 1 1 1 1 a<>good<>day<>6 5.1232 1 3 1 1 1 1 1 think<>i<>will<>6 5.1232 1 1 3 1 1 1 1 for<>testing<>i<>6 5.1232 1 1 1 3 1 1 1 testing<>i<>think<>6 5.1232 1 1 3 1 1 1 1 with<>this<>this<>7 4.8355 1 1 2 2 1 1 1 many<>things<>with<>7 4.8355 1 2 2 1 2 1 1 of<>mine<>that<>7 4.8355 1 2 2 1 2 1 1 I<>went<>to<>7 4.8355 1 2 1 2 1 1 1 things<>I<>have<>7 4.8355 1 2 2 1 1 1 1 things<>with<>this<>7 4.8355 1 2 1 2 1 1 1 mine<>that<>I<>7 4.8355 1 2 1 2 1 1 1 i<>want<>to<>8 4.4301 1 3 1 2 1 1 1 is<>a<>good<>8 4.4301 1 2 3 1 2 1 1 I<>have<>a<>8 4.4301 1 2 1 3 1 1 1 mine<>i<>want<>8 4.4301 1 2 3 1 1 1 1 this<>this<>is<>8 4.4301 1 3 2 1 1 1 1 to<>test<>many<>9 4.1424 1 1 4 2 1 1 2 many<>things<>I<>9 4.1424 1 2 2 2 2 1 1 will<>test<>many<>9 4.1424 1 1 4 2 1 1 2 i<>think<>i<>10 4.0246 1 3 1 3 1 1 1 want<>to<>test<>11 3.9193 1 1 2 5 1 1 2 went<>to<>test<>11 3.9193 1 1 2 5 1 1 2 of<>mine<>i<>12 3.7369 1 2 2 3 2 1 1 have<>a<>test<>13 3.5138 1 1 3 5 1 1 2 i<>will<>test<>13 3.5138 1 3 1 5 1 1 1 is<>a<>test<>14 2.8206 1 2 3 5 2 1 2 marimba(91): more test.tmi1 37 test<>many<>things<>1 0.4986 2 4 2 2 2 2 2 test<>of<>mine<>1 0.4986 2 4 2 2 2 2 2 this<>is<>a<>2 0.4590 2 3 2 3 2 2 2 many<>things<>with<>3 0.4286 1 2 2 1 2 1 1 of<>mine<>that<>3 0.4286 1 2 2 1 2 1 1 a<>test<>of<>4 0.4265 2 3 4 2 2 2 2 many<>things<>I<>5 0.3756 1 2 2 2 2 1 1 good<>day<>for<>6 0.3585 1 1 1 1 1 1 1 day<>for<>testing<>6 0.3585 1 1 1 1 1 1 1 of<>mine<>i<>7 0.3564 1 2 2 3 2 1 1 is<>a<>good<>8 0.3541 1 2 3 1 2 1 1 is<>a<>test<>9 0.3506 1 2 3 5 2 1 2 to<>test<>many<>10 0.3205 1 1 4 2 1 1 2 will<>test<>many<>10 0.3205 1 1 4 2 1 1 2 that<>I<>went<>11 0.3045 1 1 2 1 1 1 1 want<>to<>test<>12 0.2974 1 1 2 5 1 1 2 went<>to<>test<>12 0.2974 1 1 2 5 1 1 2 a<>good<>day<>13 0.2841 1 3 1 1 1 1 1 think<>i<>will<>13 0.2841 1 1 3 1 1 1 1 for<>testing<>i<>13 0.2841 1 1 1 3 1 1 1 testing<>i<>think<>13 0.2841 1 1 3 1 1 1 1 with<>this<>this<>14 0.2515 1 1 2 2 1 1 1 I<>went<>to<>14 0.2515 1 2 1 2 1 1 1 things<>I<>have<>14 0.2515 1 2 2 1 1 1 1 things<>with<>this<>14 0.2515 1 2 1 2 1 1 1 mine<>that<>I<>14 0.2515 1 2 1 2 1 1 1 i<>want<>to<>15 0.2323 1 3 1 2 1 1 1 I<>have<>a<>15 0.2323 1 2 1 3 1 1 1 mine<>i<>want<>15 0.2323 1 2 3 1 1 1 1 this<>this<>is<>15 0.2323 1 3 2 1 1 1 1 have<>a<>test<>16 0.2265 1 1 3 5 1 1 2 i<>think<>i<>17 0.2142 1 3 1 3 1 1 1 i<>will<>test<>18 0.1960 1 3 1 5 1 1 1 marimba(96): diff test.pmi test.pmi1 21,22d20 < test<>many<>things<>6 7.4189 2 4 2 2 2 2 2 < test<>of<>mine<>6 7.4189 2 4 2 2 2 2 2 26d23 < this<>is<>a<>7 7.2490 2 3 2 3 2 2 2 30d26 < a<>test<>of<>9 6.8339 2 3 4 2 2 2 2 34c30,34 < is<>a<>test<>11 5.5120 1 2 3 5 2 1 2 --- > test<>many<>things<>11 6.4189 2 4 2 2 2 2 2 > test<>of<>mine<>11 6.4189 2 4 2 2 2 2 2 > this<>is<>a<>12 6.2490 2 3 2 3 2 2 2 > a<>test<>of<>13 5.8339 2 3 4 2 2 2 2 > is<>a<>test<>14 5.5120 1 2 3 5 2 1 2 marimba(97): diff test.ll test.ll1 marimba(98): diff test.ps test.ps1 marimba(99): diff test.tmi test.tmi1 On Thu, Jun 11, 2009 at 8:21 AM, Ted Pedersen<duluth...@gmail.com> wrote: > Hi Gunn, > > Here's a simple test case that would be useful to see the results of... > Below I'm providing the output from the "short form" of the input name... > I'll do the same with the long form too... > > input : > > this is a test of mine i want to test many things with this > this is a good day for testing i think i will test many things > I have a test of mine that I went to test > > count.pl --ngram 3 test.cnt input > > statistic.pl --ngram 3 pmi test.pmi test.cnt > statistic.pl --ngram 3 ll test.ll test.cnt > statistic.pl --ngram 3 ps test.ps test.cnt > statistic.pl --ngram 3 tmi test.tmi test.cnt > > marimba(46): more test.pmi > 37 > good<>day<>for<>1 10.4189 1 1 1 1 1 1 1 > day<>for<>testing<>1 10.4189 1 1 1 1 1 1 1 > that<>I<>went<>2 9.4189 1 1 2 1 1 1 1 > a<>good<>day<>3 8.8339 1 3 1 1 1 1 1 > think<>i<>will<>3 8.8339 1 1 3 1 1 1 1 > for<>testing<>i<>3 8.8339 1 1 1 3 1 1 1 > testing<>i<>think<>3 8.8339 1 1 3 1 1 1 1 > with<>this<>this<>4 8.4189 1 1 2 2 1 1 1 > many<>things<>with<>4 8.4189 1 2 2 1 2 1 1 > of<>mine<>that<>4 8.4189 1 2 2 1 2 1 1 > I<>went<>to<>4 8.4189 1 2 1 2 1 1 1 > things<>I<>have<>4 8.4189 1 2 2 1 1 1 1 > things<>with<>this<>4 8.4189 1 2 1 2 1 1 1 > mine<>that<>I<>4 8.4189 1 2 1 2 1 1 1 > i<>want<>to<>5 7.8339 1 3 1 2 1 1 1 > is<>a<>good<>5 7.8339 1 2 3 1 2 1 1 > I<>have<>a<>5 7.8339 1 2 1 3 1 1 1 > mine<>i<>want<>5 7.8339 1 2 3 1 1 1 1 > this<>this<>is<>5 7.8339 1 3 2 1 1 1 1 > test<>many<>things<>6 7.4189 2 4 2 2 2 2 2 > test<>of<>mine<>6 7.4189 2 4 2 2 2 2 2 > to<>test<>many<>6 7.4189 1 1 4 2 1 1 2 > many<>things<>I<>6 7.4189 1 2 2 2 2 1 1 > will<>test<>many<>6 7.4189 1 1 4 2 1 1 2 > this<>is<>a<>7 7.2490 2 3 2 3 2 2 2 > i<>think<>i<>7 7.2490 1 3 1 3 1 1 1 > want<>to<>test<>8 7.0970 1 1 2 5 1 1 2 > went<>to<>test<>8 7.0970 1 1 2 5 1 1 2 > a<>test<>of<>9 6.8339 2 3 4 2 2 2 2 > of<>mine<>i<>9 6.8339 1 2 2 3 2 1 1 > have<>a<>test<>10 6.5120 1 1 3 5 1 1 2 > i<>will<>test<>10 6.5120 1 3 1 5 1 1 1 > is<>a<>test<>11 5.5120 1 2 3 5 2 1 2 > > 37 > good<>day<>for<>1 10.4189 1 1 1 1 1 1 1 > day<>for<>testing<>1 10.4189 1 1 1 1 1 1 1 > that<>I<>went<>2 9.4189 1 1 2 1 1 1 1 > a<>good<>day<>3 8.8339 1 3 1 1 1 1 1 > think<>i<>will<>3 8.8339 1 1 3 1 1 1 1 > for<>testing<>i<>3 8.8339 1 1 1 3 1 1 1 > testing<>i<>think<>3 8.8339 1 1 3 1 1 1 1 > with<>this<>this<>4 8.4189 1 1 2 2 1 1 1 > many<>things<>with<>4 8.4189 1 2 2 1 2 1 1 > of<>mine<>that<>4 8.4189 1 2 2 1 2 1 1 > I<>went<>to<>4 8.4189 1 2 1 2 1 1 1 > things<>I<>have<>4 8.4189 1 2 2 1 1 1 1 > things<>with<>this<>4 8.4189 1 2 1 2 1 1 1 > mine<>that<>I<>4 8.4189 1 2 1 2 1 1 1 > i<>want<>to<>5 7.8339 1 3 1 2 1 1 1 > is<>a<>good<>5 7.8339 1 2 3 1 2 1 1 > I<>have<>a<>5 7.8339 1 2 1 3 1 1 1 > mine<>i<>want<>5 7.8339 1 2 3 1 1 1 1 > this<>this<>is<>5 7.8339 1 3 2 1 1 1 1 > test<>many<>things<>6 7.4189 2 4 2 2 2 2 2 > test<>of<>mine<>6 7.4189 2 4 2 2 2 2 2 > to<>test<>many<>6 7.4189 1 1 4 2 1 1 2 > many<>things<>I<>6 7.4189 1 2 2 2 2 1 1 > will<>test<>many<>6 7.4189 1 1 4 2 1 1 2 > this<>is<>a<>7 7.2490 2 3 2 3 2 2 2 > i<>think<>i<>7 7.2490 1 3 1 3 1 1 1 > want<>to<>test<>8 7.0970 1 1 2 5 1 1 2 > went<>to<>test<>8 7.0970 1 1 2 5 1 1 2 > a<>test<>of<>9 6.8339 2 3 4 2 2 2 2 > of<>mine<>i<>9 6.8339 1 2 2 3 2 1 1 > have<>a<>test<>10 6.5120 1 1 3 5 1 1 2 > i<>will<>test<>10 6.5120 1 3 1 5 1 1 1 > is<>a<>test<>11 5.5120 1 2 3 5 2 1 2 > > marimba(48): more test.ps > 37 > test<>many<>things<>1 8.2848 2 4 2 2 2 2 2 > test<>of<>mine<>1 8.2848 2 4 2 2 2 2 2 > this<>is<>a<>2 8.0492 2 3 2 3 2 2 2 > a<>test<>of<>3 7.4739 2 3 4 2 2 2 2 > good<>day<>for<>4 6.2218 1 1 1 1 1 1 1 > day<>for<>testing<>4 6.2218 1 1 1 1 1 1 1 > that<>I<>went<>5 5.5287 1 1 2 1 1 1 1 > a<>good<>day<>6 5.1232 1 3 1 1 1 1 1 > think<>i<>will<>6 5.1232 1 1 3 1 1 1 1 > for<>testing<>i<>6 5.1232 1 1 1 3 1 1 1 > testing<>i<>think<>6 5.1232 1 1 3 1 1 1 1 > with<>this<>this<>7 4.8355 1 1 2 2 1 1 1 > many<>things<>with<>7 4.8355 1 2 2 1 2 1 1 > of<>mine<>that<>7 4.8355 1 2 2 1 2 1 1 > I<>went<>to<>7 4.8355 1 2 1 2 1 1 1 > things<>I<>have<>7 4.8355 1 2 2 1 1 1 1 > things<>with<>this<>7 4.8355 1 2 1 2 1 1 1 > mine<>that<>I<>7 4.8355 1 2 1 2 1 1 1 > i<>want<>to<>8 4.4301 1 3 1 2 1 1 1 > is<>a<>good<>8 4.4301 1 2 3 1 2 1 1 > I<>have<>a<>8 4.4301 1 2 1 3 1 1 1 > mine<>i<>want<>8 4.4301 1 2 3 1 1 1 1 > this<>this<>is<>8 4.4301 1 3 2 1 1 1 1 > to<>test<>many<>9 4.1424 1 1 4 2 1 1 2 > many<>things<>I<>9 4.1424 1 2 2 2 2 1 1 > will<>test<>many<>9 4.1424 1 1 4 2 1 1 2 > i<>think<>i<>10 4.0246 1 3 1 3 1 1 1 > want<>to<>test<>11 3.9193 1 1 2 5 1 1 2 > went<>to<>test<>11 3.9193 1 1 2 5 1 1 2 > of<>mine<>i<>12 3.7369 1 2 2 3 2 1 1 > have<>a<>test<>13 3.5138 1 1 3 5 1 1 2 > i<>will<>test<>13 3.5138 1 3 1 5 1 1 1 > is<>a<>test<>14 2.8206 1 2 3 5 2 1 2 > > marimba(51): more test.tmi > 37 > test<>many<>things<>1 0.4986 2 4 2 2 2 2 2 > test<>of<>mine<>1 0.4986 2 4 2 2 2 2 2 > this<>is<>a<>2 0.4590 2 3 2 3 2 2 2 > many<>things<>with<>3 0.4286 1 2 2 1 2 1 1 > of<>mine<>that<>3 0.4286 1 2 2 1 2 1 1 > a<>test<>of<>4 0.4265 2 3 4 2 2 2 2 > many<>things<>I<>5 0.3756 1 2 2 2 2 1 1 > good<>day<>for<>6 0.3585 1 1 1 1 1 1 1 > day<>for<>testing<>6 0.3585 1 1 1 1 1 1 1 > of<>mine<>i<>7 0.3564 1 2 2 3 2 1 1 > is<>a<>good<>8 0.3541 1 2 3 1 2 1 1 > is<>a<>test<>9 0.3506 1 2 3 5 2 1 2 > to<>test<>many<>10 0.3205 1 1 4 2 1 1 2 > will<>test<>many<>10 0.3205 1 1 4 2 1 1 2 > that<>I<>went<>11 0.3045 1 1 2 1 1 1 1 > want<>to<>test<>12 0.2974 1 1 2 5 1 1 2 > went<>to<>test<>12 0.2974 1 1 2 5 1 1 2 > a<>good<>day<>13 0.2841 1 3 1 1 1 1 1 > think<>i<>will<>13 0.2841 1 1 3 1 1 1 1 > for<>testing<>i<>13 0.2841 1 1 1 3 1 1 1 > testing<>i<>think<>13 0.2841 1 1 3 1 1 1 1 > with<>this<>this<>14 0.2515 1 1 2 2 1 1 1 > I<>went<>to<>14 0.2515 1 2 1 2 1 1 1 > things<>I<>have<>14 0.2515 1 2 2 1 1 1 1 > things<>with<>this<>14 0.2515 1 2 1 2 1 1 1 > mine<>that<>I<>14 0.2515 1 2 1 2 1 1 1 > i<>want<>to<>15 0.2323 1 3 1 2 1 1 1 > I<>have<>a<>15 0.2323 1 2 1 3 1 1 1 > mine<>i<>want<>15 0.2323 1 2 3 1 1 1 1 > this<>this<>is<>15 0.2323 1 3 2 1 1 1 1 > have<>a<>test<>16 0.2265 1 1 3 5 1 1 2 > i<>think<>i<>17 0.2142 1 3 1 3 1 1 1 > i<>will<>test<>18 0.1960 1 3 1 5 1 1 1 > > On Thu, Jun 11, 2009 at 6:27 AM, gunnlyse<gunnl...@yahoo.no> wrote: >> >> >> Hello, >> thank you for this clarification! >> >> The NSP values still do not match mine, and I see that it concerns ll, pmi, >> ps as well as tmi for trigrams. Evidently, there must be some error which >> probably lies in the observed or estimated frequencies (since all four >> measures produce different results than mine) >> >> I need to ask for two clarifications: >> >> (1) estimated frequency: The webpage/pmi file says: >> n1pp * np1p * npp1 >> m111= -------------------- >> nppp >> but the file 3D.pm says >> $m111=$n1pp*$np1p*$npp1/($nppp**2); " >> >> which I take to mean that we use, not nppp, but the exponent: >> n1pp * np1p * npp1 >> m111= -------------------- >> nppp * nppp >> If so, which one sould I really use? >> >> (2) Furthermore, let us return to the example trigram. When I compute the >> example trigram's pmi in the way I understand the code, I get the value >> -15.24452, instead of the NSP package's 6.4127. >> All the observed frequencies needed for pmi are directly available in the >> example trigram line, so the only thing that can explain diverging results >> is HOW we compute the value. >> May I therefore ask if you agree with the way I understand the code? >> >> For the trigram >> 355663266 >> at<>det<>er<>262744 7073841 9391062 5872364 1234064 647295 1064083 >> >> I compute m111 as: >>> > m111= 7073841 * 9391062 * 5872364 >>> > ------------------------- >>> > 355663266 >>> > >>> > = 1.0968417e+12 >>> > >>> > and PMI = log (262744 / 1.0968417e+12) = -15.24452 >> >>> > NSP's pmi returns (using the command line: >>> > statistic.pl --ngram 3 pmi outputfile inputfile ) >>> > produces the following line >> at<>det<>er<>1 6.4127 262744 7073841 9391062 5872364 1234064 647295 1064083 >> >> Best, >> Gunn >> >> > > > > -- > Ted Pedersen > http://www.d.umn.edu/~tpederse > -- Ted Pedersen http://www.d.umn.edu/~tpederse
