Re: [R] permutation test assumption?
Just an additional note: what I find interesting (that is an euphemism) is that a paper such as that got published on 2006 when a whole bunch of detailed papers on the same topic had been published in the past. For instance, the first I pick from the pile is by J. Romano, On the behavior of randomization tests without a group invariance assumption, JASA, 1990, 85 (411): 686-692. There are other related papers on that same issue of JASA. The relevance of same-variance assumption also shows up in permutation test textbooks (including, I think, Manly's, Good's, Noreen's, etc). Best, R. On Wed, Apr 9, 2008 at 9:11 PM, Greg Snow [EMAIL PROTECTED] wrote: A few comments, My first impression on reading that abstract was that it was complete nonsense. After thinking a bit about it and skimming the full article I decided that it was nonsense, but nonsense that is important to research and discuss (and therefore the paper is useful). Why is it nonsense? The permutation test is a test of the null hypothesis that the 2 (or k) groups are from the same distribution (or identically distributed, or exchangable). The abstract says that they looked at the type I error rate when the 2 groups had different variances or other differences. The type I error is defined when the null hypothesis is true, so computing a type I error rate when the null is by definition false does not make sense. However, statisticians often do analyses where all the assumptions are not necessarily true (is any population really distributed as a normal), but the tests are close enough. So with modern tools it is not suprising to see people doing permutation tests without understanding what they are really testing and the results may be close enough (or they might not be). The contribution of this paper is to test and see if the results are close enough or not when you use a permutation test to test the null that the means are equal when there are other differences in the groups. Their answer is that no, the results are not close enough and they suggest that if you want to test for equality of means, but not identical distributions, then don't use a permutation test. To expand on Thierry's original answer: If you are testing the correct hypotheses and doing a permutation test correctly, then You can do permutation tests on an unbalanced design and it will still be a correct test. Unbalance could affect the power, which you would want to take into account when designing a study, but does not affect the correctness of the test (when used properly). Hope this helps, -- Gregory (Greg) L. Snow Ph.D. Statistical Data Center Intermountain Healthcare [EMAIL PROTECTED] (801) 408-8111 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of João Fadista Sent: Tuesday, April 08, 2008 4:10 PM To: ONKELINX, Thierry; r-help@r-project.org Subject: Re: [R] permutation test assumption? Dear Thierry, Thanks for the reply. But as you may read in the paper http://bioinformatics.oxfordjournals.org/cgi/content/abstract/ 22/18/2244 when the sample sizes are not the same there may be an increase in the Type I error rate. Comments will be appreciated. Best regards, João Fadista De: ONKELINX, Thierry [mailto:[EMAIL PROTECTED] Enviada: ter 08-04-2008 15:27 Para: João Fadista; r-help@r-project.org Assunto: RE: [R] permutation test assumption? Dear João, You can do permutation tests on an unbalanced design. HTH, Thierry -- -- ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest Cel biometrie, methodologie en kwaliteitszorg / Section biometrics, methodology and quality assurance Gaverstraat 4 9500 Geraardsbergen Belgium tel. + 32 54/436 185 [EMAIL PROTECTED] www.inbo.be To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey -Oorspronkelijk bericht- Van: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Namens João Fadista Verzonden: dinsdag 8 april 2008 15:18 Aan: r-help@r-project.org Onderwerp: [R] permutation test assumption? Dear all, Can I do a permutation test if the number of individuals in one group is much bigger than in the other group? I searched the literature but I didin´t find any assumption that refers to this subject
Re: [R] permutation test assumption?
A few comments, My first impression on reading that abstract was that it was complete nonsense. After thinking a bit about it and skimming the full article I decided that it was nonsense, but nonsense that is important to research and discuss (and therefore the paper is useful). Why is it nonsense? The permutation test is a test of the null hypothesis that the 2 (or k) groups are from the same distribution (or identically distributed, or exchangable). The abstract says that they looked at the type I error rate when the 2 groups had different variances or other differences. The type I error is defined when the null hypothesis is true, so computing a type I error rate when the null is by definition false does not make sense. However, statisticians often do analyses where all the assumptions are not necessarily true (is any population really distributed as a normal), but the tests are close enough. So with modern tools it is not suprising to see people doing permutation tests without understanding what they are really testing and the results may be close enough (or they might not be). The contribution of this paper is to test and see if the results are close enough or not when you use a permutation test to test the null that the means are equal when there are other differences in the groups. Their answer is that no, the results are not close enough and they suggest that if you want to test for equality of means, but not identical distributions, then don't use a permutation test. To expand on Thierry's original answer: If you are testing the correct hypotheses and doing a permutation test correctly, then You can do permutation tests on an unbalanced design and it will still be a correct test. Unbalance could affect the power, which you would want to take into account when designing a study, but does not affect the correctness of the test (when used properly). Hope this helps, -- Gregory (Greg) L. Snow Ph.D. Statistical Data Center Intermountain Healthcare [EMAIL PROTECTED] (801) 408-8111 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of João Fadista Sent: Tuesday, April 08, 2008 4:10 PM To: ONKELINX, Thierry; r-help@r-project.org Subject: Re: [R] permutation test assumption? Dear Thierry, Thanks for the reply. But as you may read in the paper http://bioinformatics.oxfordjournals.org/cgi/content/abstract/ 22/18/2244 when the sample sizes are not the same there may be an increase in the Type I error rate. Comments will be appreciated. Best regards, João Fadista De: ONKELINX, Thierry [mailto:[EMAIL PROTECTED] Enviada: ter 08-04-2008 15:27 Para: João Fadista; r-help@r-project.org Assunto: RE: [R] permutation test assumption? Dear João, You can do permutation tests on an unbalanced design. HTH, Thierry -- -- ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest Cel biometrie, methodologie en kwaliteitszorg / Section biometrics, methodology and quality assurance Gaverstraat 4 9500 Geraardsbergen Belgium tel. + 32 54/436 185 [EMAIL PROTECTED] www.inbo.be To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey -Oorspronkelijk bericht- Van: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Namens João Fadista Verzonden: dinsdag 8 april 2008 15:18 Aan: r-help@r-project.org Onderwerp: [R] permutation test assumption? Dear all, Can I do a permutation test if the number of individuals in one group is much bigger than in the other group? I searched the literature but I didin´t find any assumption that refers to this subject for permutation tests. Best regards João Fadista Ph.d. student UNIVERSITY OF AARHUS Faculty of Agricultural Sciences Dept. of Genetics and Biotechnology Blichers Allé 20, P.O. BOX 50 DK-8830 Tjele Phone: +45 8999 1900 Direct: +45 8999 1900 E-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] Web: www.agrsci.org http://www.agrsci.org/ DJF now offers new degree programmes http://www.agrsci.org/content/view/full/34133 . News and news media http://www.agrsci.org/navigation/nyheder_og_presse . This email may contain information that is confidential. Any use or publication of this email without written permission from Faculty of Agricultural Sciences is not allowed. If you
[R] permutation test assumption?
Dear all, Can I do a permutation test if the number of individuals in one group is much bigger than in the other group? I searched the literature but I didin´t find any assumption that refers to this subject for permutation tests. Best regards João Fadista Ph.d. student UNIVERSITY OF AARHUS Faculty of Agricultural Sciences Dept. of Genetics and Biotechnology Blichers Allé 20, P.O. BOX 50 DK-8830 Tjele Phone: +45 8999 1900 Direct: +45 8999 1900 E-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] Web: www.agrsci.org http://www.agrsci.org/ DJF now offers new degree programmes http://www.agrsci.org/content/view/full/34133 . News and news media http://www.agrsci.org/navigation/nyheder_og_presse . This email may contain information that is confidential. Any use or publication of this email without written permission from Faculty of Agricultural Sciences is not allowed. If you are not the intended recipient, please notify Faculty of Agricultural Sciences immediately and delete this email. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] permutation test assumption?
Dear João, You can do permutation tests on an unbalanced design. HTH, Thierry ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest Cel biometrie, methodologie en kwaliteitszorg / Section biometrics, methodology and quality assurance Gaverstraat 4 9500 Geraardsbergen Belgium tel. + 32 54/436 185 [EMAIL PROTECTED] www.inbo.be To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey -Oorspronkelijk bericht- Van: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Namens João Fadista Verzonden: dinsdag 8 april 2008 15:18 Aan: r-help@r-project.org Onderwerp: [R] permutation test assumption? Dear all, Can I do a permutation test if the number of individuals in one group is much bigger than in the other group? I searched the literature but I didin´t find any assumption that refers to this subject for permutation tests. Best regards João Fadista Ph.d. student UNIVERSITY OF AARHUS Faculty of Agricultural Sciences Dept. of Genetics and Biotechnology Blichers Allé 20, P.O. BOX 50 DK-8830 Tjele Phone: +45 8999 1900 Direct: +45 8999 1900 E-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] Web: www.agrsci.org http://www.agrsci.org/ DJF now offers new degree programmes http://www.agrsci.org/content/view/full/34133 . News and news media http://www.agrsci.org/navigation/nyheder_og_presse . This email may contain information that is confidential. Any use or publication of this email without written permission from Faculty of Agricultural Sciences is not allowed. If you are not the intended recipient, please notify Faculty of Agricultural Sciences immediately and delete this email. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] permutation test assumption?
Dear Thierry, Thanks for the reply. But as you may read in the paper http://bioinformatics.oxfordjournals.org/cgi/content/abstract/22/18/2244 when the sample sizes are not the same there may be an increase in the Type I error rate. Comments will be appreciated. Best regards, João Fadista De: ONKELINX, Thierry [mailto:[EMAIL PROTECTED] Enviada: ter 08-04-2008 15:27 Para: João Fadista; r-help@r-project.org Assunto: RE: [R] permutation test assumption? Dear João, You can do permutation tests on an unbalanced design. HTH, Thierry ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest Cel biometrie, methodologie en kwaliteitszorg / Section biometrics, methodology and quality assurance Gaverstraat 4 9500 Geraardsbergen Belgium tel. + 32 54/436 185 [EMAIL PROTECTED] www.inbo.be To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey -Oorspronkelijk bericht- Van: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Namens João Fadista Verzonden: dinsdag 8 april 2008 15:18 Aan: r-help@r-project.org Onderwerp: [R] permutation test assumption? Dear all, Can I do a permutation test if the number of individuals in one group is much bigger than in the other group? I searched the literature but I didin´t find any assumption that refers to this subject for permutation tests. Best regards João Fadista Ph.d. student UNIVERSITY OF AARHUS Faculty of Agricultural Sciences Dept. of Genetics and Biotechnology Blichers Allé 20, P.O. BOX 50 DK-8830 Tjele Phone: +45 8999 1900 Direct: +45 8999 1900 E-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] Web: www.agrsci.org http://www.agrsci.org/ DJF now offers new degree programmes http://www.agrsci.org/content/view/full/34133 . News and news media http://www.agrsci.org/navigation/nyheder_og_presse . This email may contain information that is confidential. Any use or publication of this email without written permission from Faculty of Agricultural Sciences is not allowed. If you are not the intended recipient, please notify Faculty of Agricultural Sciences immediately and delete this email. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] permutation test assumption?
On Wed, 9 Apr 2008, João Fadista wrote: Dear Thierry, Thanks for the reply. But as you may read in the paper http://bioinformatics.oxfordjournals.org/cgi/content/abstract/22/18/2244 when the sample sizes are not the same there may be an increase in the Type I error rate. No, this is false and not what the reference above says. Please read more carefully. Comments will be appreciated. The basic assumption for permutation tests is exchangeability of the observations (regardless of sample sizes). The reference above describes a situation when this assumption is violated. Z Best regards, João Fadista De: ONKELINX, Thierry [mailto:[EMAIL PROTECTED] Enviada: ter 08-04-2008 15:27 Para: João Fadista; r-help@r-project.org Assunto: RE: [R] permutation test assumption? Dear João, You can do permutation tests on an unbalanced design. HTH, Thierry ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest Cel biometrie, methodologie en kwaliteitszorg / Section biometrics, methodology and quality assurance Gaverstraat 4 9500 Geraardsbergen Belgium tel. + 32 54/436 185 [EMAIL PROTECTED] www.inbo.be To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey -Oorspronkelijk bericht- Van: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Namens João Fadista Verzonden: dinsdag 8 april 2008 15:18 Aan: r-help@r-project.org Onderwerp: [R] permutation test assumption? Dear all, Can I do a permutation test if the number of individuals in one group is much bigger than in the other group? I searched the literature but I didin´t find any assumption that refers to this subject for permutation tests. Best regards João Fadista Ph.d. student UNIVERSITY OF AARHUS Faculty of Agricultural Sciences Dept. of Genetics and Biotechnology Blichers Allé 20, P.O. BOX 50 DK-8830 Tjele Phone: +45 8999 1900 Direct: +45 8999 1900 E-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] Web: www.agrsci.org http://www.agrsci.org/ DJF now offers new degree programmes http://www.agrsci.org/content/view/full/34133 . News and news media http://www.agrsci.org/navigation/nyheder_og_presse . This email may contain information that is confidential. Any use or publication of this email without written permission from Faculty of Agricultural Sciences is not allowed. If you are not the intended recipient, please notify Faculty of Agricultural Sciences immediately and delete this email. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] permutation test assumption?
But in that paper they refer to another (http://www.bepress.com/cgi/viewcontent.cgi?article=1053context=ucbbiostat) where they say that permutation distribution produces an asymptotically correct null distribution if (i) the sample sizes are equal Best regards, João De: Achim Zeileis [mailto:[EMAIL PROTECTED] Enviada: qua 09-04-2008 0:26 Para: João Fadista Cc: ONKELINX, Thierry; r-help@r-project.org Assunto: Re: [R] permutation test assumption? On Wed, 9 Apr 2008, João Fadista wrote: Dear Thierry, Thanks for the reply. But as you may read in the paper http://bioinformatics.oxfordjournals.org/cgi/content/abstract/22/18/2244 when the sample sizes are not the same there may be an increase in the Type I error rate. No, this is false and not what the reference above says. Please read more carefully. Comments will be appreciated. The basic assumption for permutation tests is exchangeability of the observations (regardless of sample sizes). The reference above describes a situation when this assumption is violated. Z Best regards, João Fadista De: ONKELINX, Thierry [mailto:[EMAIL PROTECTED] Enviada: ter 08-04-2008 15:27 Para: João Fadista; r-help@r-project.org Assunto: RE: [R] permutation test assumption? Dear João, You can do permutation tests on an unbalanced design. HTH, Thierry ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest Cel biometrie, methodologie en kwaliteitszorg / Section biometrics, methodology and quality assurance Gaverstraat 4 9500 Geraardsbergen Belgium tel. + 32 54/436 185 [EMAIL PROTECTED] www.inbo.be To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey -Oorspronkelijk bericht- Van: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Namens João Fadista Verzonden: dinsdag 8 april 2008 15:18 Aan: r-help@r-project.org Onderwerp: [R] permutation test assumption? Dear all, Can I do a permutation test if the number of individuals in one group is much bigger than in the other group? I searched the literature but I didin´t find any assumption that refers to this subject for permutation tests. Best regards João Fadista Ph.d. student UNIVERSITY OF AARHUS Faculty of Agricultural Sciences Dept. of Genetics and Biotechnology Blichers Allé 20, P.O. BOX 50 DK-8830 Tjele Phone: +45 8999 1900 Direct: +45 8999 1900 E-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] Web: www.agrsci.org http://www.agrsci.org/ DJF now offers new degree programmes http://www.agrsci.org/content/view/full/34133 . News and news media http://www.agrsci.org/navigation/nyheder_og_presse . This email may contain information that is confidential. Any use or publication of this email without written permission from Faculty of Agricultural Sciences is not allowed. If you are not the intended recipient, please notify Faculty of Agricultural Sciences immediately and delete this email. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] permutation test assumption?
On Wed, 9 Apr 2008, João Fadista wrote: But in that paper they refer to another (http://www.bepress.com/cgi/viewcontent.cgi?article=1053context=ucbbiostat) where they say that permutation distribution produces an asymptotically correct null distribution if (i) the sample sizes are equal I haven't read that paper in detail (but so do you, it seems). From looking at abstract and intro, my understanding is that they also study the behaviour of tests under weaker assumptions than exchangeability. Z Best regards, João De: Achim Zeileis [mailto:[EMAIL PROTECTED] Enviada: qua 09-04-2008 0:26 Para: João Fadista Cc: ONKELINX, Thierry; r-help@r-project.org Assunto: Re: [R] permutation test assumption? On Wed, 9 Apr 2008, João Fadista wrote: Dear Thierry, Thanks for the reply. But as you may read in the paper http://bioinformatics.oxfordjournals.org/cgi/content/abstract/22/18/2244 when the sample sizes are not the same there may be an increase in the Type I error rate. No, this is false and not what the reference above says. Please read more carefully. Comments will be appreciated. The basic assumption for permutation tests is exchangeability of the observations (regardless of sample sizes). The reference above describes a situation when this assumption is violated. Z Best regards, João Fadista De: ONKELINX, Thierry [mailto:[EMAIL PROTECTED] Enviada: ter 08-04-2008 15:27 Para: João Fadista; r-help@r-project.org Assunto: RE: [R] permutation test assumption? Dear João, You can do permutation tests on an unbalanced design. HTH, Thierry ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest Cel biometrie, methodologie en kwaliteitszorg / Section biometrics, methodology and quality assurance Gaverstraat 4 9500 Geraardsbergen Belgium tel. + 32 54/436 185 [EMAIL PROTECTED] www.inbo.be To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey -Oorspronkelijk bericht- Van: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Namens João Fadista Verzonden: dinsdag 8 april 2008 15:18 Aan: r-help@r-project.org Onderwerp: [R] permutation test assumption? Dear all, Can I do a permutation test if the number of individuals in one group is much bigger than in the other group? I searched the literature but I didin´t find any assumption that refers to this subject for permutation tests. Best regards João Fadista Ph.d. student UNIVERSITY OF AARHUS Faculty of Agricultural Sciences Dept. of Genetics and Biotechnology Blichers Allé 20, P.O. BOX 50 DK-8830 Tjele Phone: +45 8999 1900 Direct: +45 8999 1900 E-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] Web: www.agrsci.org http://www.agrsci.org/ DJF now offers new degree programmes http://www.agrsci.org/content/view/full/34133 . News and news media http://www.agrsci.org/navigation/nyheder_og_presse . This email may contain information that is confidential. Any use or publication of this email without written permission from Faculty of Agricultural Sciences is not allowed. If you are not the intended recipient, please notify Faculty of Agricultural Sciences immediately and delete this email. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
