Paul D. Buck wrote:
>
> On Sep 30, 2009, at 6:03 AM, Martin wrote:
>
>> OK, this is where Paul's apparent wishes (note, should be expressed as
>> "ideas",) and my ideas diverge.
>>
>>
>> Paul is proposing that special "calibration WUs" are to be passed
>> through the Boinc system end-to-end for the dual purpose of calibrating
>> the performance of the client that processed the WU, and to also act as
>> a validation check of the entire Boinc WU data path.
>>
>>
>> My proposal is that we just do the minimum necessary to calibrate the
>> host performance and credits against a known project lab reference
>> computer using the normal pool of live WUs. Then, the only 'wasted'
>> compute time is that needed to characterise the one (or few) reference
>> computer systems in the lab. Everything else is then compared against
>> them. The calibration is propagated hierarchically through the
>> participants hosts in a similar way to what is done for such as NTP and
>> for NIST standards. At least a small level of WU redundancy is required
>> so that the calibration can propagate by comparing the hosts that have
>> processed the same WU. The coordination for this can be done totally
>> server-side (as part of the validator?).
>>
>>
>> Sorry Paul, the end-to-end 'validation' is something that is likely so
>> specific to each project that it is up to the project to test/prove the
>> correctness of the Boinc generated results.
>
> The point being is that if the client is not returning good results for
> SaH, it is not that likely that it is returning good results to Rosetta.
That depends on what the 'errors' might be and how. Agreed, that is
likely to be the case, but not always. Hence the projects must have
robust tests and validation for themselves for /their/ processing and
analysis.
There is also the problem of how you can guarantee that a combined
fault-test-and-calibration WU will actually exercise all faults paths to
be a thorough test. Otherwise, by your own argument, what can we call
"thorough enough"?... Anything less than 100% fault coverage is useless?
I'm also working from the premise that it is unrealistic to try to
abstractly characterise or to instrument accurately enough live WUs to
be useful. Hence, don't waste developer time trying that. Instead, use
the real world representative example that /is/ a computer system that
can then be readily benchmarked in the lab in any way that might be chosen.
Aside: A numerical test WU could be useful as a test case during
development to highlight the differences that can be expected across
different host platforms. A sort of numerical recipes test example.
Note: I don't think that would be of any use to use as a 'special'
calibration WU. That's not needed unless you want to claim some
benchmark for the reference _computer_
[...]
> The difficulty of the extension from your suggestion to mine is so
> minimal that this is why when I started with your thought as expressed
> above I made the natural extension.
However, that leads to a very different point of reference. I don't
think that we can realistically contrive pan-dimensional project
fault-test-and-calibration WUs that would be any more accurate than the
existing whetstone-dhrystone benchmarking. There will always be another
project that uses the host hardware /differently/ ...
Hence, use the hardware itself as your ("etalon" ;-) "golden") reference
standard.
[...]
> no science value what-so-ever ... I'm sorry ... but I love irony ...
Good to see that you have some sort of humour :-)
Occam's Razor is a good idea. Try to avoid feature overload! Keep it as
simple as possible. Simple calibration should be enough. Other methods
are available and are better for validation and test of whatever WU
results are wanted.
Regards,
Martin
--
--------------------
Martin Lomas
m_boincdev ml1 co uk.ddSPAM.dd
--------------------
_______________________________________________
boinc_dev mailing list
[email protected]
http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev
To unsubscribe, visit the above URL and
(near bottom of page) enter your email address.
