All of the statistical tests have been corrected so they should now produce the correct P-Value.
An updated caret distribution, v5.616, is now available for download from http://brainvis.wustl.edu/wiki/index.php/Caret:Download (username=Rabbit password=Carrot). The effect is generally very minimal (2/iterations -- .0004 using our typical 5k). On 11/08/2010 09:18 AM, Donna Dierker wrote: > Julia, > > I see what you mean. Based on the report you uploaded, the p-values > listed here seem off by .002: > > 1 2.500 1712 1229.060425 1242.687012 -27.858 > -75.308 -13.617 0.036000 > 1 2.500 1960 1059.480713 1062.812378 -21.474 > -62.266 50.506 0.044000 > > I'll ask John about it. > > Donna > > On 11/08/2010 04:07 AM, Julia Bender wrote: >> Hi Donna, >> >> thanks for your answers. I've uploaded >> "T-Map_LH_cCue_EndoLeft.metric_TMap_Significant_Clusters.txt" which >> is the >> example file we've been talking about. As far as I see it the cluster >> >> TMap >> ---- >> Column Thresh Num-Nodes Area Area-Corrected >> COG-X COG-Y COG-Z P-Value >> 1 2.500 1712 1229.060425 1242.687012 -27.858 >> -75.308 -13.617 0.036000 >> >> fits in between rank 37 (area corrected 1273.066772) and 38 >> (area-corrected 1240.362915). >> >> Thanks for your help, >> >> Julia >> >> >> >> >>> On 11/04/2010 09:57 AM, Julia Bender wrote: >>> >>>> Hi Donna, >>>> >>>> thank you so much for your detailed reply. It helped a great deal. >>>> Three things remain somewhat unclear to me: >>>> >>>> 1. I think I figured out why the list of permuted clusters in >>>> xx.metric_TMap_Significant_Clusters.txt only has 696 instead of 1000 >>>> (=number of iterations) columns in the example contrast. My input >>>> columns >>>> were n=13, which would allow up to 2^13=8192 iterations. But only >>>> 696 of >>>> the 1000 iterations produce clusters with a T-Value above my defined >>>> threshold. Theses are the ones listed in the output file. Is that >>>> possible? >>>> >>>> >>> Sure, it certainly is possible for the data to not survive threshold, >>> but I'm not used to it. >>> >>>> 2. The significant cluster in the example has an area-corrected >>>> value of >>>> 1242.687012 and is assigned a p-value of 0.036. This means that it >>>> resides >>>> on rank 36 out of my 1000 iterations, right? When I looked up the 100 >>>> largest clusters list (my predefined P-threshold was .1) in my output >>>> file >>>> the significant cluster would actually take rank 38. what am I getting >>>> wrong here? >>>> >>>> >>> Keep in mind that the p-values are based on the distribution built by >>> the random tmaps, but the area of clusters on the real tmap is >>> typically >>> in between two areas on the randomized list. >>> >>> If your real tmap cluster is bigger than the 38th biggest random >>> cluster, but smaller than the 37th, then I'd expect it to have a >>> p-value >>> of 38/iterations. >>> >>> If this is not happening, I don't know why. Upload your significance >>> report, and I'll have a look: >>> >>> http://pulvinar.wustl.edu/cgi-bin/upload.cgi >>> >>>> 3.The correction for multiple comparisons is done by thresholding all >>>> real >>>> T-map clusters above the predefined T-Value with the smallest >>>> iterations*alpha cluster of the permutation distribution according to >>>> area-corrected? The corrected P-value for real T-map clusters that >>>> survive >>>> this threshold is derived from the rank the clusters hold in the total >>>> iterations list according to area-corrected? >>>> >>>> >>> I would put it differently. Any clusters in the real tmap surviving >>> the >>> threshold that exceed the minimum significance cut-off in corrected >>> area >>> are significant. >>> >>>> Again, thanks a lot for your answers!! >>>> >>>> Julia >>>> >>>> >>>> >>>>> Hi Julia, >>>>> >>>>> Bear with me, because it's been years since I used the caret_command >>>>> tests, which are mostly cluster tests. We switched from cluster to >>>>> TFCE >>>>> a year or so ago: >>>>> >>>>> http://brainvis.wustl.edu/wiki/index.php/Caret:Documentation:Statistics#Threshold-Free_Cluster_Enhancement_.28TFCE.29 >>>>> >>>>> >>>>> >>>>> But those tests are in caret_stats, which is a separate tool/package >>>>> based on java. Let me know if you want to know more about that. >>>>> >>>>> My inline replies below reflect my best recollection of the cluster >>>>> tests. >>>>> >>>>> Donna >>>>> >>>>> On 10/08/2010 08:16 AM, Julia Bender wrote: >>>>> >>>>> >>>>>> Hi Donna, >>>>>> >>>>>> I've looked more into the "caret_command >>>>>> -metric-statistics-one-sample-t-test" output files. I'm having >>>>>> trouble >>>>>> understanding what all the information means. Maybe you can correct >>>>>> me: >>>>>> >>>>>> 1. xx.metric_TMap.metric: >>>>>> Map of T-Values for the wholebrain (?) for the contrast defined >>>>>> in the >>>>>> xx.metric input files. Those are T-Values exceeding the T-Value >>>>>> thresholds >>>>>> and alpha level I specified in >>>>>> "caret_command-metric-statistics-one-sample-t-test" (eg -300000 2.5 >>>>>> 0.1 >>>>>> in >>>>>> my case). >>>>>> >>>>>> >>>>> They are not thresholded. The 2.5 and 0.1 affect the cluster size, >>>>> which affects downstream outputs. But this TMap is unthresholded. >>>>> >>>>> >>>>>> They are not corrected for multiple comparisons(?). >>>>>> >>>>>> >>>>> No, definitely not. >>>>> >>>>> >>>>>> This is the one I should load onto my surface. When I load >>>>>> that TMap, why can I still see T-Values below my defined threshold? >>>>>> >>>>>> >>>>>> >>>>> Right. I like to use the unthresholded t/f-map for my figures, but >>>>> generate a border around the clusters that were significant, and show >>>>> the border overlaid on the unthresholded t-map. >>>>> >>>>> >>>>>> 2.xx.metric_ShuffledTMap.metric: >>>>>> Distribution of T-Values derived from permuting + and - on each >>>>>> element >>>>>> in >>>>>> xx in N iterations (in my case 1000) >>>>>> >>>>>> >>>>>> >>>>> Right, and less than 1000, depending on how many columns there are in >>>>> your input composite metric. >>>>> >>>>> >>>>>> 3.xx.metric_TMap_Significant_Clusters.txt: >>>>>> This is what the help page says: >>>>>> "1. Find the biggest cluster in each column of the permutation T-Map >>>>>> metric/shape file and sort them by cluster size." >>>>>> I see two lists of clusters in the output file. I assume the one >>>>>> that >>>>>> is >>>>>> the result of this sorting is the lower one. It has about 700 rows >>>>>> depending on xx, why does it not have 1000 rows, one for each >>>>>> permutation? >>>>>> >>>>>> >>>>>> >>>>> See Sample Report: Two Sample T-Test here: >>>>> >>>>> http://brainvis.wustl.edu/OLD/courses/stats_neurosci/2008_BMEcourse/BME_dld_talk.htm >>>>> >>>>> >>>>> >>>>> Yours is a one-sample t-test, but I think the cluster lists will >>>>> be the >>>>> same. >>>>> >>>>> If your last list of clusters has less than 1000 rows, then you had >>>>> fewer than 10 columns in your input composite metric. If n is the >>>>> number of columns, and 2 raised to the n is less than your input >>>>> iterations, then Caret will stop at 2^n. >>>>> >>>>> >>>>>> Are the clusters sorted by Num-Nodes, Area or Area corrected? >>>>>> >>>>>> >>>>> Descending area-corrected. >>>>> >>>>> >>>>>> In my files >>>>>> they seem kind of sorted by both... How is a cluster defined? >>>>>> >>>>>> >>>>>> >>>>> By your input thresholds. The permuted t-maps are thresholded at the >>>>> level specified, and then clusters of contiguous supra-threshold >>>>> nodes >>>>> are found. Only the largest in each iteration is saved. Then >>>>> they are >>>>> listed in descending order of area-corrected size. >>>>> >>>>> >>>>>> "2. Find the largest (alpha)(iterations) clusters in the Permutation >>>>>> T-Map >>>>>> and use its cluster size as the Significant Cluster Cutoff." >>>>>> I assume this is the list of T-Values right below the above list of >>>>>> clusters, why does it contain clusters with a P-Value that is >>>>>> above my >>>>>> defined alpha? >>>>>> >>>>>> >>>>>> >>>>> The last table lists all the largest clusters for each iteration, >>>>> regardless of its p. The second table is what you want. >>>>> >>>>> >>>>>> "3. Find clusters in the Real T-Map file." This must be the upper >>>>>> list >>>>>> of >>>>>> clusters (containing much less rows than the lower one) >>>>>> >>>>>> >>>>>> >>>>> Correct. That link above shows this the best. >>>>> >>>>> >>>>>> "4. Report all clusters in Real T-Map file that are larger than >>>>>> Significant Cluster Cutoff." This is the list of T-Values below this >>>>>> list, >>>>>> containing only clusters that are bigger than the cluster with the >>>>>> highest >>>>>> P-Value found in 2. that pass the alpha and T-Value thresholds I >>>>>> specified >>>>>> in "caret_command-metric-statistics-one-sample-t-test". >>>>>> >>>>>> >>>>>> >>>>> Again, here is the relevant excerpt from the link above: >>>>> >>>>> Significant Area: 306.226 <--- area of smallest cluster listed in >>>>> next >>>>> section >>>>> >>>>> Shuffled TMap <--- Top alpha*iterations biggest clusters are listed >>>>> below, >>>>> in descending area-corrected sequence. >>>>> (i.e., the smallest of which determines the signifiance area cut-off) >>>>> ------------- >>>>> Column Thresh Num-Nodes Area Area-Corrected COG-X >>>>> COG-Y COG-Z P-Value >>>>> 821 2.660 2300 1439.637207 1559.143188 -37.249 >>>>> 0.007 -3.456 >>>>> 150 2.660 792 598.519104 858.216492 -49.142 >>>>> -32.369 11.918 >>>>> 548 2.660 1380 641.563843 643.790466 -35.004 >>>>> 2.519 -14.981 >>>>> ... (middle biggest alpha*iterations entries omitted) >>>>> 681 -2.660 279 249.739059 312.872681 -48.189 >>>>> -9.576 10.352 >>>>> 649 2.660 237 198.468857 311.602844 -16.170 >>>>> -79.223 30.149 >>>>> 633 -2.660 217 119.337837 306.226257 -42.479 >>>>> -48.846 41.505 >>>>> >>>>> >>>>> TMap <--- Significant real tmap clusters (i.e., >= significant area) >>>>> are >>>>> listed here; no entries here means no clusters were significant. >>>>> ---- >>>>> Column Thresh Num-Nodes Area Area-Corrected COG-X >>>>> COG-Y COG-Z P-Value >>>>> 3 2.660 344 229.353973 362.322113 -45.677 >>>>> -27.442 18.039 0.029000 >>>>> >>>>> Shuffled TMap <--- All iterations max clusters are listed below, in >>>>> descending area-corrected sequence. >>>>> ------------- >>>>> Column Thresh Num-Nodes Area Area-Corrected COG-X >>>>> COG-Y COG-Z P-Value >>>>> 821 2.660 2300 1439.637207 1559.143188 -37.249 >>>>> 0.007 -3.456 >>>>> 150 2.660 792 598.519104 858.216492 -49.142 >>>>> -32.369 11.918 >>>>> 548 2.660 1380 641.563843 643.790466 -35.004 >>>>> 2.519 -14.981 >>>>> ... (middle biggest alpha*iterations entries omitted) >>>>> 3 2.660 1 0.475079 0.472114 -18.520 >>>>> -24.418 -21.858 0.999000 >>>>> 3 -2.660 1 0.474738 0.456814 -21.478 >>>>> -36.579 -17.077 0.999000 >>>>> 3 2.660 1 0.000000 0.000000 -6.822 >>>>> -46.751 8.581 0.999000 >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> >>>>> >>>>> >>>>>> 4. T-Map_LH_cCue_EndoL_vs_Cue_ExoL.metric_TMapClusters.metric >>>>>> Is this the map of clusters defined above? >>>>>> >>>>>> >>>>>> >>>>> This is what I believe you thought xx.metric_TMap.metric was, but >>>>> it is >>>>> unthresholded. >>>>> >>>>> This TMapClusters one zeroes out all nodes that are NOT within a >>>>> significant cluster. >>>>> >>>>> >>>>>> I'm sorry, I know these are a many questions. Thanks a lot for your >>>>>> help! >>>>>> >>>>>> Julia >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Julia, >>>>>>> I looked at your report and your t-map, which is consistent with >>>>>>> the >>>>>>> >>>>>>> >>>>>>> >>>>>> caret_command -metric-information output you included below. >>>>>> >>>>>> >>>>>> >>>>>>> Just making sure you understand this part of the report: >>>>>>> TMap >>>>>>> ---- >>>>>>> Column Thresh Num-Nodes Area Area-Corrected COG-X >>>>>>> >>>>>>> >>>>>>> >>>>>> COG-Y COG-Z P- >>>>>> >>>>>> >>>>>> >>>>>>> Value >>>>>>> 1 2.500 3064 2223.228027 2245.943848 -30.317 >>>>>>> -73.796 -12.228 0.0 >>>>>>> 12000 >>>>>>> 1 2.500 3372 1865.999878 1863.883423 -21.516 >>>>>>> -62.842 46.382 0.0 >>>>>>> 17000 >>>>>>> 1 2.500 1557 681.308838 674.981384 -32.967 >>>>>>> -5.873 48.701 0.0 >>>>>>> 59000 >>>>>>> These are the clusters in your real t-map that were significant at >>>>>>> the >>>>>>> >>>>>>> >>>>>>> >>>>>> 0.1 >>>>>> >>>>>> >>>>>> >>>>>>> alpha you specified, using the 2.5 threshold. (Note that all the >>>>>>> >>>>>>> >>>>>>> >>>>>> significant clusters were at the positive end.) >>>>>> >>>>>> >>>>>> >>>>>>> I believe the reason you saw different max/min in the Caret GUI was >>>>>>> that >>>>>>> >>>>>>> >>>>>>> >>>>>> you had the permuted t-map loaded, instead of the real one. In your >>>>>> message below, you said, "Adjustment:Column: permuted >>>>>> T-Values,Threshold >>>>>> type". There is nothing about permuted in the file you >>>>>> uploaded. If >>>>>> you >>>>>> >>>>>> >>>>>> >>>>>>> were viewing the permuted/shuffled t-map, this would also >>>>>>> explain why >>>>>>> >>>>>>> >>>>>>> >>>>>> little would survive a low threshold. >>>>>> >>>>>> >>>>>> >>>>>>> But we don't necessarily (or even usually) use the same values we >>>>>>> used >>>>>>> >>>>>>> >>>>>>> >>>>>> for >>>>>> >>>>>> >>>>>> >>>>>>> cluster thresholds as the threshold for displaying t-maps, e.g., >>>>>>> for >>>>>>> >>>>>>> >>>>>>> >>>>>> publication purposes. I think we like to see some color >>>>>> differentiation >>>>>> beyond the cluster threshold max. If they are the same, the color >>>>>> will >>>>>> saturate at the max. Sometimes we'll use a p-value derived from the >>>>>> degrees of freedom and get a corresponding t-value from that, and >>>>>> use >>>>>> that >>>>>> >>>>>> >>>>>> >>>>>>> for thresholding. Other times we might just use, say, +/-4.0 or >>>>>>> higher, >>>>>>> >>>>>>> >>>>>>> >>>>>> depending on how big the values get in the data. Usually we'll >>>>>> use a >>>>>> symmetric scale (i.e., -x to +x -- rather than different min/max). >>>>>> Donna >>>>>> >>>>>> >>>>>> >>>>>>> On 08/11/2010 09:12 AM, Julia Bender wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Hi Donna, >>>>>>>> I've just uploaded the two files. >>>>>>>> Thanks for your help! >>>>>>>> Julia >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Julia, >>>>>>>>> It will be easier for me to get my head around your question if I >>>>>>>>> can >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>> get two files: >>>>>> >>>>>> >>>>>> >>>>>>>>> * T-Map_LH_cCue_Endo.metric_TMap.metric (whatever the final >>>>>>>>> output >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>> metric was, but NOT the permuted/shuffled tmap file). >>>>>> >>>>>> >>>>>> >>>>>>>>> * The report named something like *Signicance*.txt >>>>>>>>> Could you upload those here: >>>>>>>>> http://pulvinar.wustl.edu/cgi-bin/upload.cgi >>>>>>>>> My brain would be ever so grateful. >>>>>>>>> Donna >>>>>>>>> On 08/11/2010 07:07 AM, Julia Bender wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Hey everyone, >>>>>>>>>> I'm a bit confused about how to threshold my T-Maps in caret5. I >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> created >>>>>> >>>>>> >>>>>> >>>>>>>>>> the maps with the following command: >>>>>>>>>> /usr/local/caret/bin_linux/caret_command >>>>>>>>>> -metric-statistics-one-sample-t-test $EACHMETRIC $FIDUCIAL_COORD >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> $OPEN_TOPO $SURFACE_SHAPE 3 T-Map_$EACHMETRIC -300000.0 2.5 0.10 >>>>>> 10 1 >>>>>> 1000 >>>>>> >>>>>> >>>>>> >>>>>>>>>> 0 4 >>>>>>>>>> So I put the negative threshold to -300000 and the positive >>>>>>>>>> threshold >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> to >>>>>> >>>>>> >>>>>> >>>>>>>>>> 2.5. When I look at the resulting Tmap.metric files it gives me >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> something >>>>>> >>>>>> >>>>>> >>>>>>>>>> like this: >>>>>>>>>> Filename: T-Map_LH_cCue_Endo.metric_TMap.metric >>>>>>>>>> Number of Nodes: 73730 >>>>>>>>>> Number of Columns: 1 >>>>>>>>>> Column Minimum Maximum Mean Sample Dev >>>>>>>>>> % >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> Positive % Negative Column Name >>>>>> >>>>>> >>>>>> >>>>>>>>>> 1 -9.785 6.076 -0.950 2.639 >>>>>>>>>> 36.234 63.766 T-Values >>>>>>>>>> As far as I understand, this means the maximum negative T >>>>>>>>>> value in >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> this >>>>>> >>>>>> >>>>>> >>>>>>>>>> metric is -9.785 and the maximum positive T value is 6.076. >>>>>>>>>> When I >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> open >>>>>> >>>>>> >>>>>> >>>>>>>>>> the file in caret though (Color mapping: Auto Scale, Display >>>>>>>>>> mode: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> Both, >>>>>> >>>>>> >>>>>> >>>>>>>>>> Display Color Bar, Threshold Adjustment:Column: permuted >>>>>>>>>> T-Values,Threshold type: Column) the bar tells me that my >>>>>>>>>> maximum >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> negative >>>>>> >>>>>> >>>>>> >>>>>>>>>> value is -3.7 and my maximum positive value is 2.6. This also >>>>>>>>>> holds >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> when >>>>>> >>>>>> >>>>>> >>>>>>>>>> I >>>>>>>>>> adjust the thresholds in the fields below to -2 and 2, when >>>>>>>>>> almost >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> all >>>>>> >>>>>> >>>>>> >>>>>>>>>> activation disappears. >>>>>>>>>> Which is the correct information? >>>>>>>>>> Thanks a lot for your help! >>>>>>>>>> Julia >>>>>>>>>> Dipl. Psych. Julia Bender >>>>>>>>>> Humboldt Universität zu Berlin >>>>>>>>>> Mathematisch - Naturwissenschaftliche Fakultät II >>>>>>>>>> Institut für Psychologie, Abt. Klinische Psychologie >>>>>>>>>> Unter den Linden 6 >>>>>>>>>> D-10099 Berlin >>>>>>>>>> >>>>>>>>>> >>> _______________________________________________ >>> caret-users mailing list >>> [email protected] >>> http://brainvis.wustl.edu/mailman/listinfo/caret-users >>> >>> >> >> >> Dipl. Psych. Julia Bender >> Humboldt Universit�t zu Berlin >> Mathematisch - Naturwissenschaftliche Fakult�t II >> Institut f�r Psychologie, Abt. Klinische Psychologie >> Unter den Linden 6 >> D-10099 Berlin >> >> _______________________________________________ >> caret-users mailing list >> [email protected] >> http://brainvis.wustl.edu/mailman/listinfo/caret-users >> > > _______________________________________________ caret-users mailing list [email protected] http://brainvis.wustl.edu/mailman/listinfo/caret-users
