OK. You have a case where you sample from a 'population' of times from situation A a
number of times, and from Situation B a number of times. Maybe C, D, etc. too.
To compare 2 of these babies, use a t test. Keep sample size (n's of each) about
equal, and go for it. Student 't' test is pretty darn robust. Means test relies on
Central Limit Theorem, etc. Depending on how much work it takes to run a simulation,
you might do this 20 times each, to get a pretty fair discriminatory ability.
to compare more than 2, say A through E, a one way AoV would work. IF the respective
distributions were Normal, and the variances were reasonably close together. Have to
check this out first. If not, you have some choices. One is to back off the
confidence in your conclusion, and select the 'best' condition for further study. If
the 'best' one is 'obvious' this may get you on to the next step. Otherwise, some
(possibly effective) modifications of AoV may correct the deviations from assumptions.
And I haven't said anything about testing for Power, but I suspect you are not up to
that yet. Patience :)
this is a pretty quick way to do it. Depending on how rigorous you want to be, it
could be more than enough.
Jay
Gooseman wrote:
Hi,
Thanks for your help! I went over these ideas and now understand my
problem better.
If I explain my simulation, it may help. Basically, I have a
simulation where various agents have to find a target. The
simulation is terminated once the target has been found. The current
measurement of performance is iterations taken - time. The
simulation settings are kept constant, aside from the starting
positions which are random. The simulation is repeated until the
confidence interval reaches a certain percentage [this statement may
be wrong once you read then next step!]
The simulation then changes a parameter (such as number of agents)
and is then repeated to sample the new population. This is done quite
a few times.
What I really need to do, is to prove with a certain confidence, that
the MEAN time taken from Simulation A comes from a different
population than from Simulation B, C, D, E etc.
From my undestating, this may imply that I need to concentrate on the
accuracy of the mean of simulation run wrt the real population mean
(unknown) and then compare this to other simulation runs with
different populations. Some suggestions have included doing a ANOVA
analysis. Comparing multi variances was also suggested, but this
apparenly can only be done with 2 populations.
On top of this, there is a big requirement on computational efficiency
- each simulation needs to stop when the results are accurate enough
for the next step. So is confidence in the mean the solution (and how
do I do that), or is it comparing various simulation runs together
(and using what method) or is it something else, or a combination.
Does this explain enough? If anyone requires any more info, just ask.
Sorry if this explanation or question sounds vague - I am just
starting to find my way around stats!
Many thanks!
[EMAIL PROTECTED] (Jay Warner) wrote in message news:[EMAIL PROTECTED]...
the real question is, 'how much accuracy (precision, variance) is
suitable?'
If you were to repeat the simulation run (i.e., a test) a total of n
times, then you could say that the true mean elapsed time was x-bar +/-
(certain amount), with say 95% confidence.
That is, if you were to then repeat the whole process, n times again, 95%
of the time the x-bar would fall within the +/- (certain amount) you had
calculated. The average of your mean elapsed time is probably Normal, so
this equation can be used. If you want to predict the one next elapsed
time from the next simulation run, then you have to believe that your
individual times are Normally distributed, or do some deeper analysis.
If that's confusing, I'm sorry, but it comes from what you asked.
You can do the simulation run n times, and _estimate_ a value for mean
elapsed time that could be confirmed only by say 100*n runs. Does this
sound like what you want?
The eq. for the 'certain amount' is given by
certain amount = s*z/sqrt(n)
where s = stdev of your n run times, z = 1.96 for 95% confidence, and n =
number of simulation runs.
Pick a confidence interval ('certain amount') that you like, then solve
for n to decide how many runs you will need to make. Statistics cannot
tell you what confidence interval is suitable to your problem - that is a
technical issue. It can tell you now many n's you need to reach that
confidence interval.
Is this what you were looking for?
Cheers,
Jay
PS:Yes, I know 'accuracy' and 'precision' refer to different things.
But you used the first of these words in a way which I infer meant the
latter, so I opened the first sentence in that manner.
Gooseman wrote:
Hi,
I am writing a computer simulation, and I