Hi,
I thought part of the point was that it sometimes gives false
positives. A minimal perfect hash function is a bijective mapping
h : Keys -> [0, |Keys|)
But if you give it a phrase that isn't a key (and not in the domain of
the function), it makes up an integer. CMPH does not protect against
keys outside the map nor should it.
The common trick to protect against keys outside the map is to
fingerprint each key, storing fingerprint f(k) in an array indexed by
h(k). Of course, fingerprints can collide, so the false positive
probability is 1/2^b where b is the number of bits used in the
fingerprint. The value of b is a trade-off between size and
false-positive probability.
Kenneth
On 11/19/12 10:49, Rico Sennrich wrote:
> Hi Marcin,
>
> I got a false positive with a compact phrase table (Friday's git
> revision 4813df). I can't find any hint in the documentation about the
> format being probabilistic, so I'm assuming it's a bug. Here's how I
> compacted the model:
>
> ~/mosesdecoder/bin/processPhraseTableMin
> -in /path/to/model/phrase-table.gz -out /path/to/model/phrase-table
> -threads 12 -alignment-info -encoding None
>
> I don't know if you'll be able to reproduce it (with my compacted model,
> it's reproducible, but I don't know if cmph is deterministic), but
> here's the model (2MB):
>
> https://docs.google.com/open?id=0B5nFI-wykGgndFF6aFQycEZmVnc
>
>
>
> sennrich@home ~$ echo "to the European Community" |
> mosesdecoder/bin/queryPhraseTableMin -t phrase-table
>
> to the European Community ||| rechts von Ersetzung ||| 0.5 0.00872443 1
> 0.000228467 2.718
>
> "to the European Community" does not exist in the phrase table, and this
> phrase pair seems to be used:
>
> to the right of Replacement ||| rechts von Ersetzung ||| 0.5 0.00872443
> 1 0.000228467 2.718 ||| 1-0 2-0 3-1 4-2 ||| 2 1 1
>
>
>
> best wishes,
> Rico
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
