Hello, I am trying to prune my phrase table using the SALM generated suffix array indices but I don't see anything pruned. I am using the news-commentary-v10 data for english to french translation.
I followed the instructions on this wiki: http://www.statmt.org/moses/?n=Advanced.RuleTables#ntoc5 Here is the command I used to run this: I generated the suffix arrays using: SALM/Bin/Linux/Index/IndexSA.O32 TARGET SALM/Bin/Linux/Index/IndexSA.O32 SOURCE [sbhadour@sbhadour-ld1 model]$ cat phrase-table | /home/sbhadour/work/experiments/mosesdecoder/contrib/sigtest-filter/filter-pt -e /home/sbhadour/work/experiments/lang-training-data/salm-sufix-arr-index/en_fr/news-commentary-v10.fr-en.clean.fr.sa_suffix -f /home/sbhadour/work/experiments/lang-training-data/salm-sufix-arr-index/en_fr/news-commentary-v10.fr-en.clean.en.sa_suffix -l 'a+e -n 30' > phrase-table.pruned -l = a+e -n 30 Filtering using P(e|f) only. n=0 ..................................................[n:500000] ..................................................[n:1000000] ..................................................[n:1500000] ..................................................[n:2000000] ..................................................[n:2500000] ..................................................[n:3000000] ..................................................[n:3500000] ..................................................[n:4000000] ..................................................[n:4500000] ..................................................[n:5000000] ..................................................[n:5500000] ..................................................[n:6000000] ............ ------------------------------------------------------ unfiltered phrases pairs: 6121925 P(f|e) filter [first]: 0 (0%) significance filter: 0 (0%) TOTAL FILTERED: 0 (0%) FILTERED phrase pairs: 6121925 (100%) ------------------------------------------------------ Am I missing anything? Help appreciated! -- Regards, Sameer Bhadouria.
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
