Thanks to all those who took part in my Flash speed test a few weeks ago. I'd
like to share back the results, but I must caveat that I only got a small
amount of data, so these results are not statically significant.
The Purpose
=
I wanted to know how consistent the timing is in Flash across the plethora
of web browsers out there in the wild. I had been working on a app that
somewhat reliest on consistent timing, hence my curiosity.
The Method
=
I wrote a very simple app using the flex framework, that did the following:
1. Set the framerate to x
2. Measure the system time (getTimer())
3. Add an event listener to Event.ENTER_FRAME
4. on each frame enter, increment a counter, until the counter == x * 10
5. Measure the system time again, and deduct the value from (2)
6. Compute the implied framerate as (x * 10)/ value from (5)
7. Capture OS/Browser setup from UserAgent string, using JS.
8. send data to server (GAE)
I did this for x=24 and for x=1000
Known Experimental Deficiencies
=
a) getTimer may not be accurate
b) incrementing a counter once per frame is not representative of the real
computational load most apps would place per frame
c) individual system performance varies with time, e.g. with the load placed
by other apps, with GC, etc
d) I don't know enough about the internals on the Flash player, or the
browsers, to really know what's going on in there, and what else might make
this experiment flawed (unknown unknowns...)
e) Some of the data points came from the same PCs (including mine)
f) Most tests were done on developers' machines, which are not
representative of the average machine
g) I didn't test anything between x=24 and x=1000, and there may be speeds
between 24 and 1000 that operate very differently.
Results
==
Data Set:
data points (n) = 134
n By OS - Mac: 28, Win: 100, Linux: 6
n By x - x=1000 FPS: 73,x=24 FPS: 61
Range of results:
x=24: min=16.5, max=24.2
x=1000: min=24.3, max=998.9
Results by OS/browser:
(the data gets spotty here, so I'll summarize)
x=1000
OS = Mac
n= 10
results: min = 24.3, max=58.0, mean = 49.6
results: 7 @ 56-58, 2 @ 24-26, 1...@46
x=24
OS = Mac
n= 18
results: min = 16.5, max=24.1, mean = 21.5
Safari 4 on Intel: mean=23.9, n=4
Safari 4 on PPC: mean=16.5, n=1
Safari 3 on Intel: mean=19.4, n=3
FF3.0.z on Intel: mean = 20.6, n=5
(other data points are one-offs, so not quoted)
x=1000
OS = Win XP / Vista (excluded Win 2000, and 'Windows')
n= 56
results: min = 54.6, max=998.9, mean = 586
FF 3.0.8: mean = 930, n=21, min=780, max=998.9
IE 6.z 7.z 8.z: mean = 62.6, n=13, min=54.7 max=64.1
Chrome: mean = 187.3, n=5, min=173.5 max = 199.7
x=24
OS = Win XP / Vista (excluded Win 2000, and 'Windows')
n= 40
results: min = 20.7, max=24.2, mean =23.1
FF 3.0.8: mean =23.3 , n=17, min=21.6, max=24.2
IE 6.z 7.z 8.z: mean =23.1 , n=11, min=20.7 max=24.2
Chrome: mean =24.2 , n=2, min=24.169 max =24.174
x=1000
OS=Linux
n=3
mean=80.9 ,min=66.8, max=105
x=24
OS=Linux
n=3
mean = 24.0, min=23.8, max=24.1
Discussion
None of this data is statistically significant, as there isn't enough of it,
but a few patterns emerge nonetheless. To prevent me peppering this section
with qualifying words, like 'maybe', 'perhaps', etc I shall present these
observations as facts, though they clearly ARE NOT.
1) You can't get higher than 60 FPS on Mac when you request 1000 FPS
2) Requesting 24 FPS on a Mac gives varied results. Safari 4 seems pretty
reliable, unless the machine is running PowerPC. FF 3.0.z delivers a mean on
20.6 FPS, which 14% slow of the target 24 FPS
3) Windows is very different to Mac
4) The max FPS on windows is highly dependent on the browser.
5) The actual FPS on windows when FPS 1000 is requested for a specific
browser is pretty consistent - FF 3.0.8 is clustered around 900, Chrome is
clustered around 180, IE is clustered around 60
6) The actual FPS on Windows when FPS 24 is requested is pretty reliable on
all browsers, averaging 23.1 FPS
7) The actual FPS on Windows and Mac when requesting FPS 24 never exceeded
24.2 FPS - i.e. you 'never' get a speed that's too fast.
To summarize these point observations: don't rely on actually getting the
FPS that you request, and don't rely on what you do get being consistent
across browsers and OSs. It appears that FPS 24 is a fairly safe bet (but
FPS 40 might be too, for all I know). If you can, write apps
that don't require the FPS to be correct for the app to function properly
(i.e. dont try to derive time from a count of ENTER_FRAME,
either implicitly or explicitly). Test extensively across browsers and OSs,
especially cross Windows-Mac.
I'd be interested to hear what you all think - what important things have I
missed?
cheers
tom
