Nick Craig-Wood <[EMAIL PROTECTED]> wrote:
>Some time ago I proposed a standard format for save files. It also
>had the advantage that it only saved the actual bits used so would
>make the save files 1MByte rather than 1.8 MByte for an 8,000,000
>exponent in prime95.
Richard Crandall has defined such a standard format for Fermat number
residue files - this helped immensely with the Pepin test of F24. For
example, during the roughly 7 months of the test I maintained an archive
of Pepin residues for every iteration == 0 mod 4096, i.e. we wound up
with a total of around 2^24/2^12 = 2^12 of such files, each about 2MB
in size. When Jason Papadopoulos' floating-point-based test finished
(a few days after mine) he just grabbed my final squaring file and diff-ed
it with the one produced by his code to verify that our final residues
were the same. Similarly, at the same time our two machines were doing
their respective floating-point tests, Richard was overseeing a separate
all-integer check. Since the all-integer algorithms tend to run 5-10X
slower on similar hardware as the floating-FFT-based ones, the only way
to effect an all-integer verification was to parallelize it: a floating-
point run serves as a "wavefront" and deposits residues periodically, and a
number of "drone" machines running integer-based convolutions, with each
grabbing a pair of residue files, say drone Y gets files for squarings x1
and x2, with x1 < x2. Drone Y then starts with the x1 residue, does (x2-x1)
modular all-integer squarings, then compares that result with the contents
of the floating-point generated x2 residue file. In this way, one can use
multiple slow machines to check the results of one fast machine in about
the same time the fast machine needs for the calculation. Having a standard,
platform-independent savefile format means that one can write a high-level
(e.g. C) code for the verification, compile it on any needed platform, and
not have to worry about converting savefile formats. Crandall's Fermat-number
format does a bytewise write of the residue, i.e. one doesn't need to worry
about whether a given machine uses a big or little-endian scheme for storing
individual bytes of integer data.
>At the moment I think both Prime95 and the unix programs write the floating
>point values of the FFT array to a file which is inherently non-portable
>across machine architectures. It also makes the save files quite a bit
>larger than necessary (an important consideration if we ever want to hold a
>central repository of these files).
Not true - all of the fastest codes (Prime95, Mlucas and MacLucasUNIX) write
residue digits in 4-byte integer form to one or more savefiles. Only MLU
writes a floating-point version of the residue, in addition to the integer.
The two easiest ways to make a platform-independent savefile format are
(i) bytewise, as above, or (ii) in 4-byte integer form, with a preset
-endian convention. The latter might be faster to read and write, if
e.g. a machine that uses a big-endian register storage convention can
still read/write little-endian integers from/to disk as fast as it can
do for big-endian. Or one can do bytewise (which always works in C - my
f90-based Fermat number code used a small C module for this I/O) and
simply buffer the reads and writes into chunks of the same size used by
the target platform for disk I/O (4096-byte blocks are common, but one
could define the platform-dependent blocksize as a parameter.
I discussed the idea of a platform-independent savefile format with George
and Scott several times during the past year, in conjunction with the idea
of central (e.g. PrimeNet) storage of user residues at periodic intervals;
Scott's major objection was that he didn't think the amount of work lost
due to hard disk problems, accidental savefile deletion or user dropout
justifies such an effort. On the other hand, he admitted he didn't really
have a good estimate of the incidence of the above events (except possibly
the latter item). My main point was that as exponents get larger, the
likelihood of the above occurrences is likely to increase, but one still
needs a reasonable estimate of how frequently it occurs now. The best
prospect for doing this would seem to be some kind of e-mail poll of the
current users, and that needs a decent response rate to be significant.
(And human memories are notoriously unreliable.)
A less intrusive, logistically simpler method might be to simply look at
the percentage of exponents that are reassigned by PrimeNet after their
time limit expires. One might assume that that these were on average
20% completed, so if 5% of exponents get reassigned, one estimates that
1% of work could be saved if the savefiles for said exponents could be
reassigned, as well.
Interestingly, Scott seemed to think the bandwidth needed for storing
and managing user savefiles was less of a problem than justifying the
need for such a strategy in the first place. (Scott, correct me if I'm wrong
in my recollection about this.) Obviously not all users
will have sufficiently fast Internet connections to allow them to
participate in such a scheme, but it's not something that should be
mandatory in any event.
-Ernst
_________________________________________________________________
Unsubscribe & list info -- http://www.scruz.net/~luke/signup.htm
Mersenne Prime FAQ -- http://www.tasam.com/~lrwiman/FAQ-mers
