Hi,
just for the records, recompiling prune-lm with additionally setting the
compiler option
-fno-strict-aliasing
solved the problem. It seems gcc 4.1.2 didn't like the magic casting
that's used in some of the source files.
Is there a place where these kind of issues are documented?
best regards
Christof
Christof Pintaske wrote:
> Hi,
>
> I created a 3-gram LM with the irstlm toolkit (5.0.22). The LM has about
> 25M entries:
>
> ngram 1= 300209
> ngram 2= 4864097
> ngram 3= 20336549
>
>
> I tried to prune it with prune-lm on a Linux machine.
>
> prune-lm --threshold=1e-6,1e-6 sun.irstlm.gz sun.pruned.irlstlm &> x.out
>
> In the out x.out I get repeated error messages
>
> ng: qu 0 ts=1.00059 tbs=0.0196106 k=0 ns=20
>
> probably more than 100M identical ones. After running the pruning over
> night the stderr output reached 100GB size and I stopped the process.
>
> Just looking at the source code I assume that lmtable::wdprune() loops
> endless over the "prune:" goto statement. Are there any problems with
> the pscale() routine?
>
> Any hints where to look at are highly appreciated.
>
> best regards
> Christof
>
> _______________________________________________
> Moses-support mailing list
> [email protected]
> http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
