Graeme Geldenhuys wrote:
On 2013-02-11 19:03, Mark Morgan Lloyd wrote:
No of tests | Type of Tests | Linux | FreeBSD
-------------+-----------------+------------+----------------
151 | CSV persistence | 0:22 | 0:27
-------------+-----------------+------------+----------------
151 | TAB persistence | 0:22 | 0:27
-------------+-----------------+------------+----------------
151 | XMLLight | 0:23 | 0:26
-------------+-----------------+------------+----------------
151 | SqlDB-Firebird | 3:14 | 3:38
-------------+-----------------+------------+----------------
682 | Non-Persistent | 1:09 | 1:30
-------------+-----------------+------------+----------------
As you can see, consistently the FreeBSD tests take longer than the
Linux ones. The test project on each platform was compiled with exactly
the same compiler settings.
What exactly are we looking at there: 151 iterations inside a single
program, or 151 programs?
The unit testing project is a single executable running all the above
tests. 151 is 151 different unit tests to test the various persistence
layers. The test suite is based on a hierarchy of classes, that is why
all persistence tests have 151. The exact same persistence tests are run
for each persistence layer. The 682 is again different tests for
anything non persisting - testing various classes, and pretty much all
functionality of those classes.
I did not setup the testing framework to run multiple iterations, only
one run was completed with a total of around 1200+ unit tests taking
around 4:30-5:00 to complete, from start to finish.
So memory cache etc should really have an effect. Because each test case
starts from scratch, does the test, then cleans up. Then the next test
etc etc.
There's a vast number of factors that you're underreporting. For
example, the underlying VM system could detect which OS is running as a
guest, and behave differently. Or the OS could reconfigure the CPU and
chipset (specifically, the cache architecture) to a different extent. Or
Linux's habit of using all available memory as disc buffering could be
accelerating startup and/or execution. Or BSD could be making heavy
going of fixing up dynamic library references. Or the different guests
could be running their (simulated) disc controllers differently.
The only way that you're going to get anywhere is by tweaking the
programs to loop, so that you can factor startup time out. Then separate
groups of tests that don't make heavy OS use from those that do. All
basic engineering stuff.
--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk
[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel
