Re: [R] statistical test for comparison of two classifications (nominal)
Martin, Pardon the delayed reply. Bootstrap methods have been around for some time (late seventies?), but their popularity seems to have exploded in correspondence with computing technology. You should be able to find more information in most modern books on statistical inference, but here is a brief: The bootstrap is a method often used to establish an empirical null distribution for a test statistic when traditional (analytical) methods fail. The bootstrap works by imposing a null hypothesis on the observed data, followed by re-sampling with replacement. The test statistic is computed at each re-sample and used to build up an empirical null distribution. The idea is to impose the null hypothesis while preserving variability in the observed data, and thus the test statistic. For example, suppose we observe some continuous scalar data and hypothesize that the sample was observed from a population with mean zero. We can impose this hypothesis by subtracting the sample mean from each observation. Re-samples from these transformed data are treated as having been observed under the null hypothesis. In the case of classification and partitioning, the difficulty is formulating a meaningful null hypothesis about the collection of classifications, and imposing the null hypothesis in a bootstrap sampling scheme. -Matt On Wed, 2010-11-17 at 10:01 -0500, Martin Tomko wrote: Thanks Mat, I have in the meantime identified the Rand index, but not the others. I will also have a look at profdpm, that did not pop-up in my searches. Indeed, the interpretation is going to be critical... Could you please elaborate on what you mean by the bootstrap process? Thanks a lot for your helps, Martin On 11/17/2010 3:50 PM, Matt Shotwell wrote: There are several statistics used to compare nominal classifications, or _partitions_ of a data set. A partition isn't quite the same in this context because partitioned data are not restricted to a fixed number of classes. However, the statistics used to compare partitions should also work for these 'restricted' partitions. See the Rand index, Fowlkes and Mallows index, Wallace indices, and the Jaccard index. The profdpm package implements a function (?profdpm::pci) that computes these indices for two factors representing partitions of the same data. The difficult part is drawing statistical inference about these indices. It's difficult to formulate a null hypothesis, and even more difficult to determine a null distribution for a partition comparison index. A bootstrap test might work, but you will probably have to implement this yourself. -Matt On Wed, 2010-11-17 at 08:33 -0500, Martin Tomko wrote: Dear all, I am having a hard time to figure out a suitable test for the match between two nominal classifications of the same set of data. I have used hierarchical clustering with multiple methods (ward, k-means,...) to classify my dat into a set number of classesa, and I would like to compare the resulting automated classification with the actual - objective benchmark one. So in principle I have a data frame with n columns of nominal classifications, and I want to do a mutual comparison and test for significance in difference in classification between pairs of columns. I just need to identify a suitable test, but I fail. I am currently exploring the possibility of using Cohen's Kappa, but I am open to other suggestions. Especially the fact that kappa seems to be moslty used on failible, human annotators seems to bring in limitations taht do not apply to my automatic classification. Any help will be appreciated, especially if also followed by a pointer to an R package that implements it. Thanks Martin __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Matthew S. Shotwell Graduate Student Division of Biostatistics and Epidemiology Medical University of South Carolina __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] statistical test for comparison of two classifications (nominal)
Dear all, I am having a hard time to figure out a suitable test for the match between two nominal classifications of the same set of data. I have used hierarchical clustering with multiple methods (ward, k-means,...) to classify my dat into a set number of classesa, and I would like to compare the resulting automated classification with the actual - objective benchmark one. So in principle I have a data frame with n columns of nominal classifications, and I want to do a mutual comparison and test for significance in difference in classification between pairs of columns. I just need to identify a suitable test, but I fail. I am currently exploring the possibility of using Cohen's Kappa, but I am open to other suggestions. Especially the fact that kappa seems to be moslty used on failible, human annotators seems to bring in limitations taht do not apply to my automatic classification. Any help will be appreciated, especially if also followed by a pointer to an R package that implements it. Thanks Martin __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] statistical test for comparison of two classifications (nominal)
There are several statistics used to compare nominal classifications, or _partitions_ of a data set. A partition isn't quite the same in this context because partitioned data are not restricted to a fixed number of classes. However, the statistics used to compare partitions should also work for these 'restricted' partitions. See the Rand index, Fowlkes and Mallows index, Wallace indices, and the Jaccard index. The profdpm package implements a function (?profdpm::pci) that computes these indices for two factors representing partitions of the same data. The difficult part is drawing statistical inference about these indices. It's difficult to formulate a null hypothesis, and even more difficult to determine a null distribution for a partition comparison index. A bootstrap test might work, but you will probably have to implement this yourself. -Matt On Wed, 2010-11-17 at 08:33 -0500, Martin Tomko wrote: Dear all, I am having a hard time to figure out a suitable test for the match between two nominal classifications of the same set of data. I have used hierarchical clustering with multiple methods (ward, k-means,...) to classify my dat into a set number of classesa, and I would like to compare the resulting automated classification with the actual - objective benchmark one. So in principle I have a data frame with n columns of nominal classifications, and I want to do a mutual comparison and test for significance in difference in classification between pairs of columns. I just need to identify a suitable test, but I fail. I am currently exploring the possibility of using Cohen's Kappa, but I am open to other suggestions. Especially the fact that kappa seems to be moslty used on failible, human annotators seems to bring in limitations taht do not apply to my automatic classification. Any help will be appreciated, especially if also followed by a pointer to an R package that implements it. Thanks Martin __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Matthew S. Shotwell Graduate Student Division of Biostatistics and Epidemiology Medical University of South Carolina __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] statistical test for comparison of two classifications (nominal)
Thanks Mat, I have in the meantime identified the Rand index, but not the others. I will also have a look at profdpm, that did not pop-up in my searches. Indeed, the interpretation is going to be critical... Could you please elaborate on what you mean by the bootstrap process? Thanks a lot for your helps, Martin On 11/17/2010 3:50 PM, Matt Shotwell wrote: There are several statistics used to compare nominal classifications, or _partitions_ of a data set. A partition isn't quite the same in this context because partitioned data are not restricted to a fixed number of classes. However, the statistics used to compare partitions should also work for these 'restricted' partitions. See the Rand index, Fowlkes and Mallows index, Wallace indices, and the Jaccard index. The profdpm package implements a function (?profdpm::pci) that computes these indices for two factors representing partitions of the same data. The difficult part is drawing statistical inference about these indices. It's difficult to formulate a null hypothesis, and even more difficult to determine a null distribution for a partition comparison index. A bootstrap test might work, but you will probably have to implement this yourself. -Matt On Wed, 2010-11-17 at 08:33 -0500, Martin Tomko wrote: Dear all, I am having a hard time to figure out a suitable test for the match between two nominal classifications of the same set of data. I have used hierarchical clustering with multiple methods (ward, k-means,...) to classify my dat into a set number of classesa, and I would like to compare the resulting automated classification with the actual - objective benchmark one. So in principle I have a data frame with n columns of nominal classifications, and I want to do a mutual comparison and test for significance in difference in classification between pairs of columns. I just need to identify a suitable test, but I fail. I am currently exploring the possibility of using Cohen's Kappa, but I am open to other suggestions. Especially the fact that kappa seems to be moslty used on failible, human annotators seems to bring in limitations taht do not apply to my automatic classification. Any help will be appreciated, especially if also followed by a pointer to an R package that implements it. Thanks Martin __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Martin Tomko Postdoctoral Research Assistant Geographic Information Systems Division Department of Geography University of Zurich - Irchel Winterthurerstr. 190 CH-8057 Zurich, Switzerland email: martin.to...@geo.uzh.ch site: http://www.geo.uzh.ch/~mtomko mob:+41-788 629 558 tel:+41-44-6355256 fax:+41-44-6356848 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] statistical test for comparison of two classifications (nominal)
On Nov 17, 2010, at 7:33 AM, Martin Tomko wrote: Dear all, I am having a hard time to figure out a suitable test for the match between two nominal classifications of the same set of data. I have used hierarchical clustering with multiple methods (ward, k-means,...) to classify my dat into a set number of classesa, and I would like to compare the resulting automated classification with the actual - objective benchmark one. So in principle I have a data frame with n columns of nominal classifications, and I want to do a mutual comparison and test for significance in difference in classification between pairs of columns. I just need to identify a suitable test, but I fail. I am currently exploring the possibility of using Cohen's Kappa, but I am open to other suggestions. Especially the fact that kappa seems to be moslty used on failible, human annotators seems to bring in limitations taht do not apply to my automatic classification. Any help will be appreciated, especially if also followed by a pointer to an R package that implements it. Thanks Martin In addition to Matt's comments, you might want to consider marginal homogeneity tests. There are extensions of the pairwise McNemar test to greater than two categories. Some online information is here: http://www.john-uebersax.com/stat/mcnemar.htm and there is the ?mh_test implemented in the 'coin' package on CRAN. HTH, Marc Schwartz __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] statistical test for comparison of two classifications (nominal)
Another useful measure to compare partitions is the adjusted Rand index which is implemented in the library(e1071) within the classAgreement function. If you have your data partitions to be compared in a matricial form (where each column is a different partition), the syntax is ARI-classAgreement(table(data[,i],data[,j]))$crand Other useful measures of goodness-of-fit for clustering are the silhouette index or the c-index or the Goodman-Kruskal index. although they evaluate in general inter/intra-cluster distance distributions. For instance, you can maximise/minimise these indices to find the best partition among a set of candidate ones. Mattia Prosperi. 2010/11/17 Marc Schwartz marc_schwa...@me.com: On Nov 17, 2010, at 7:33 AM, Martin Tomko wrote: Dear all, I am having a hard time to figure out a suitable test for the match between two nominal classifications of the same set of data. I have used hierarchical clustering with multiple methods (ward, k-means,...) to classify my dat into a set number of classesa, and I would like to compare the resulting automated classification with the actual - objective benchmark one. So in principle I have a data frame with n columns of nominal classifications, and I want to do a mutual comparison and test for significance in difference in classification between pairs of columns. I just need to identify a suitable test, but I fail. I am currently exploring the possibility of using Cohen's Kappa, but I am open to other suggestions. Especially the fact that kappa seems to be moslty used on failible, human annotators seems to bring in limitations taht do not apply to my automatic classification. Any help will be appreciated, especially if also followed by a pointer to an R package that implements it. Thanks Martin In addition to Matt's comments, you might want to consider marginal homogeneity tests. There are extensions of the pairwise McNemar test to greater than two categories. Some online information is here: http://www.john-uebersax.com/stat/mcnemar.htm and there is the ?mh_test implemented in the 'coin' package on CRAN. HTH, Marc Schwartz __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] statistical test for comparison of two classifications (nominal)
Thank you Matta for the great suggestion, I will try the additional tests. I have just been experimenting with the e1071 package and the adjustedRand. It works perfectly, The only outstadning question is interpretation - is there any rule of thumbs for the level of agreement that needs to be reached in order to say there is High Agreement or similar? Thanks Martin On 11/17/2010 4:49 PM, Mattia Prosperi wrote: Another useful measure to compare partitions is the adjusted Rand index which is implemented in the library(e1071) within the classAgreement function. If you have your data partitions to be compared in a matricial form (where each column is a different partition), the syntax is ARI-classAgreement(table(data[,i],data[,j]))$crand Other useful measures of goodness-of-fit for clustering are the silhouette index or the c-index or the Goodman-Kruskal index. although they evaluate in general inter/intra-cluster distance distributions. For instance, you can maximise/minimise these indices to find the best partition among a set of candidate ones. Mattia Prosperi. 2010/11/17 Marc Schwartzmarc_schwa...@me.com: On Nov 17, 2010, at 7:33 AM, Martin Tomko wrote: Dear all, I am having a hard time to figure out a suitable test for the match between two nominal classifications of the same set of data. I have used hierarchical clustering with multiple methods (ward, k-means,...) to classify my dat into a set number of classesa, and I would like to compare the resulting automated classification with the actual - objective benchmark one. So in principle I have a data frame with n columns of nominal classifications, and I want to do a mutual comparison and test for significance in difference in classification between pairs of columns. I just need to identify a suitable test, but I fail. I am currently exploring the possibility of using Cohen's Kappa, but I am open to other suggestions. Especially the fact that kappa seems to be moslty used on failible, human annotators seems to bring in limitations taht do not apply to my automatic classification. Any help will be appreciated, especially if also followed by a pointer to an R package that implements it. Thanks Martin In addition to Matt's comments, you might want to consider marginal homogeneity tests. There are extensions of the pairwise McNemar test to greater than two categories. Some online information is here: http://www.john-uebersax.com/stat/mcnemar.htm and there is the ?mh_test implemented in the 'coin' package on CRAN. HTH, Marc Schwartz __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
