An NSP user has the following idea: -------------- I just thought it would be nice to have an option in NSP (specifically in statistic.pl) to filter bigrams based on their p-values, like we currently do by rank and score. Very often I need to find "significant" bigrams, and it will be nice if I can just tell NSP to give me bigrams with say < 5% or 1% chance of being independent etc... --------------
I think this is an excellent suggestion, and it is something we have thought about doing in the past, and should revisit now that we are back into NSP development mode (just started in Sept, hopefully some new releases coming in October!). In fact, there are some Perl modules that will give these values (assuming we can give them log-likelihood or pearson's values, which of course we can). For example, the following seems promising: http://search.cpan.org/~mikek/Statistics-Distributions-1.02/Distributions.pm So, we aren't too far away from being able to do this, especially for measures like ll and x2 (which can be assigned significance based on their raw values using 1 degree of freedom and the chi-squared distribution). However, the problem we would have is that it's less clear what it means for some measures - like the dice coefficient, for example. I don't *think* there is a clean way to assign significance to those values (perhaps I'm wrong on that point?) Anyway, an excellent suggestion. Thanks for making it - we'll make sure we give it some serious consideration as we get into modifying statistic.pl, which isn't too far off in the future. If there are other suggestions along these lines, that is additional features for statistic.pl, additional measures to support, etc. please make them now as we are early in the development stages and it's a good time to add items to the agenda. A few months from now it will probably be a bit harder to do so. Thanks! Ted ------------------------ Yahoo! Groups Sponsor --------------------~--> Get Bzzzy! (real tools to help you find a job). Welcome to the Sweet Life. http://us.click.yahoo.com/A77XvD/vlQLAA/TtwFAA/dpFolB/TM --------------------------------------------------------------------~-> Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/ngram/ <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
