Hi Rico, fixed. I missed a type of decompression inconsistency caused by false positives (hopefully the last one). Interestingly, this bug has not appeared for months with tens or hundreds of phrase tables here at WIPO. A few hours after you reported it, we, too, encountered the same issue for the first time with one of our phrase tables. Thank you for that!
Best, Marcin W dniu 22.11.2012 11:05, Rico Sennrich pisze: > Marcin Junczys-Dowmunt <junczys@...> writes: > >> Hi Rico, >> cmph is not probabilistic, but the check for phrases outside the set of >> training phrases (false positives) is. For encodings others than None, >> the compression also serves as a test, so the fp rate is much smaller >> then. You can also modifiy the fp probability by increasing the number >> of fingerprint bits, for instance to 20 (-fingerprint 20), default is >> 16. The probability of a FP slipping through per query is 1/2^b where b >> is the number of fingerprint bits. Using b=16 and -encoding PREnc >> (default) I have never seen a false positive, so just stick to the >> defaults :) > Hi Marcin, > > sorry, I should have seen that false positives are not a bug per se. The > problem > was just that they broke my code, and apparently your code also sometimes > segfaults when encountering them. > > here are the steps to reproduce the bug (looking up "various authorities" > causes > the segfault; it's not in the table) > > =================== > command executed: > > echo "various authorities" | gdb "~/mosesdecoder/bin/queryPhraseTableMin -t > table -n 3 -a" > > =================== > resources used: > table.minphr (compiled on 64bit Linux), hope you can use it > > https://docs.google.com/open?id=0B5nFI-wykGgnUEttRTNlRk94Z3c > > =================== > backtrace: > > #0 AddWord (newWord=..., this=0x22e1550) at moses/Phrase.h:131 > #1 Moses::Phrase::Append (this=0x22e1550, endPhrase=...) at > moses/Phrase.cpp:138 > #2 0x0000000000594471 in Moses::PhraseDecoder::DecodeCollection > (this=0x228d490, tpv=..., encodedBitStream=..., sourcePhrase=..., > topLevel=true) > at moses/CompactPT/PhraseDecoder.cpp:412 > #3 0x0000000000593657 in Moses::PhraseDecoder::CreateTargetPhraseCollection > (this=0x228d490, sourcePhrase=..., topLevel=true) at > moses/CompactPT/PhraseDecoder.cpp:245 > #4 0x00000000004ac0b8 in > Moses::PhraseDictionaryCompact::GetTargetPhraseCollectionRaw > (this=0x7fff9694c560, sourcePhrase=...) at > moses/CompactPT/PhraseDictionaryCompact.cpp:158 > #5 0x0000000000409749 in main (argc=<optimized out>, argv=<optimized out>) at > misc/queryPhraseTableMin.cpp:75 > > > > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
