I thought it was clear that what we want to establish a difference between two (say players X and Y) by running games and testing that gammon-rate(X) != gammon-rate(Y).
-Joseph On Wed, 13 Nov 2019 at 10:39, Timothy Y. Chow <[email protected]> wrote: > On Wed, 13 Nov 2019, Joseph Heled wrote: > > "but for most practical purposes this is an irrelevant technicality" > > > > Are you saying that I can treat each of the 4 estimates independently? > > That is, use sqrt(pq/N) as the std for each? seems problematic to me :) > > No, I didn't say that. As I said, the 4 estimates are not independent. > What I recommended was for you to compute the sample standard deviation > for each parameter of interest. So for example, if you have 100 samples > and you're interested in the gammon rate, then first compute the mean > gammon rate over all your samples. Call that mu. Then for each sample > value g_i, compute (g_i - mu)^2. Sum these up, divide by 100, and take > the square root. This will give you some indication of the dispersion of > your sample set. > > The formula sqrt(pq/N) arises when you're doing hypothesis testing. It's > the standard deviation under the null hypothesis. But so far, you haven't > specified a null hypothesis. > > > Yes, a Bayesian approach would be better, but this probably involves > > things like contour integration or other horrors. > > No, it doesn't. But you do need to specify a prior distribution. > Suppose you're interested in the win rate, and your prior distribution is > uniform on the interval [0,1]. For illustration purposes, let's say > you're satisfied with accuracy to 1 decimal place, so each of the > probabilities in the set {0.1, 0.2, ..., 0.9} has prior probability 1/9. > Now you start to collect data. Say the first data point is a win. Then > using Bayes's rule, you find that the posterior probability of a win rate > of j/10 is obtained by multiplying the prior probability by j/10, and then > normalizing so that everything sums to 1. So the posterior probabilities > work out to be > > [1/45, 2/45, 3/45, 4/45, 5/45, 6/45, 7/45, 8/45, 9/45] > > Similarly, if you observe a loss, then you adjust by multiplying the prior > probability by 1 - j/10 and normalizing. Repeat for every observation in > your sample. > > Tim >
