On Tue, Apr 19, 2011 at 06:09:09PM +0100, Daniel Drake wrote:
> Maybe dependent on the possibility of an extX-specific optimization in
> the .zd file, ...
Here's another optimisation for review as well. It isn't quite
extX-specific though.
On a 253MB .zd4 file the fs-update took three minutes on Toshiba Class 4
SD, compared to the usual 15 minutes. It took four minutes on the usual
SanDisk SD.
This was a test build of 10.2 containing only Sugar and six activities.
This five-fold reduction in fs-update would not occur for your current
builds, but there would be some considerable reduction anyway,
especially on larger SD card images where the ratio of unused blocks to
used blocks is higher.
The algorithm is:
- create a psuedo-random marker block of 4096-bytes, called a fill
pattern,
- instead of creating a sparse image file, create the image file by
appending enough of the fill pattern blocks to reach the requested
size, ... on a fast system this is an extra few seconds,
- during zhashfs, skip processing of zblocks that contain only the fill
pattern, and for those that contain the fill pattern in only part of
the zblock, zero it so that the fill pattern is not used as input to
the compression.
The visual effect during fs-update is wonderful. The display skips
forward, leaving skipped blocks grey, and begins writing green blocks.
It clearly shows how the space is allocated.
There is one small problem left to fix; Q3A64 fs-update displays an
error at the end because the total number of blocks written has not
agreed with the number of blocks specified at the top of the file. I
propose to fix that somehow, but I've yet to decide on exactly how. It
was introduced by -r2186, where on your request Mitch changed the check
from last block written to number of blocks written. It does not happen
with Q3A62.
The written filesystem images pass "e2fsck -f" when run from a USB
booted Linux, and the laptop boots and runs normally. An "rpm -Va"
looks normal, when the olpc-os-builder timestamp changes are filtered
out.
The attached patch is to olpc-os-builder, unfortunately to an instance I
had lying around gathering dust. Good enough for testing the idea.
It's from 27th July 2010, but the change looks easy to apply to master.
--
James Cameron
http://quozl.linux.org.au/
diff --git a/bin/zhashfs.c b/bin/zhashfs.c
index 7f7bcc0..4525f30 100644
--- a/bin/zhashfs.c
+++ b/bin/zhashfs.c
@@ -6,6 +6,9 @@
#include "zlib.h"
#define DO(x) do { run_cmd((x), __LINE__, __FILE__, #x); } while (0);
+
+#define PATTERN_SIZE 4096
+
void run_cmd(int res, int line, char *file, char *cmd)
{
if (res != CRYPT_OK) {
@@ -31,13 +34,18 @@ int main(int argc, char **argv)
int hashid, readlen;
int j;
- int allf;
+ int skip;
int zresult;
FILE *zfile;
uLongf zlen;
unsigned char *p;
unsigned char *zbuf; // ZBUFSIZE
+ unsigned char *pbuf; // fill pattern buffer
+ char *pname; // fill pattern file name
+ FILE *pfile; // fill pattern file
+ int patterned, n;
+
if (argc < 6) {
fprintf(stderr, "%s: zblocksize hashname signed_file_name spec_file_name zdata_file_name [ #blocks ]\n", argv[0]);
return EXIT_FAILURE;
@@ -71,6 +79,16 @@ int main(int argc, char **argv)
infile = fopen(argv[3], "rb");
LTC_ARGCHK(infile != NULL);
+ /* open and read the fill pattern file */
+ pname = strcat(argv[3], ".fill");
+ pfile = fopen(pname, "rb");
+ LTC_ARGCHK(pfile != NULL);
+ pbuf = malloc(PATTERN_SIZE);
+ LTC_ARGCHK(pbuf != NULL);
+ n = fread(pbuf, 1, PATTERN_SIZE, pfile);
+ LTC_ARGCHK(n == PATTERN_SIZE);
+ fclose(pfile);
+
/* open output file */
outfile = fopen(argv[4], "wb");
LTC_ARGCHK(outfile != NULL);
@@ -113,18 +131,35 @@ int main(int argc, char **argv)
LTC_ARGCHK(readlen == zblocksize);
#ifdef notdef
- allf = 1;
+ skip = 1;
for (p = (unsigned char *)buf; p < &buf[zblocksize]; p++) {
if (*p != 0xff) {
- allf = 0;
+ skip = 0;
break;
}
}
#else
- allf = 0;
+ skip = 0;
#endif
+ /* check if this zblock is fully patterned as unused, and if
+ any parts of the zblock are patterned then zero them, for ease
+ of compression */
+
+ patterned = 1;
+ for (n = 0; n < (zblocksize / PATTERN_SIZE); n++) {
+ if (memcmp(&buf[n*PATTERN_SIZE], pbuf, PATTERN_SIZE)) {
+ patterned = 0;
+ } else {
+ memset(&buf[n*PATTERN_SIZE], 0, PATTERN_SIZE);
+ }
+ }
+
+ /* skip any block that is fully patterned, thus relying on the
+ fs-update card erase-blocks */
+
+ if (patterned) skip++;
- if (!allf) {
+ if (!skip) {
mdlen = sizeof(md);
DO(hash_memory(hashid, buf, zblocksize, md, &mdlen));
diff --git a/modules/sd_card_image/image.50.makefs.sh b/modules/sd_card_image/image.50.makefs.sh
index f78b7ee..52474e3 100644
--- a/modules/sd_card_image/image.50.makefs.sh
+++ b/modules/sd_card_image/image.50.makefs.sh
@@ -40,7 +40,25 @@ make_image()
local img=$outputdir/os$buildnr.disk.img
fi
- dd if=/dev/zero of=$img bs=$BLOCK_SIZE count=0 seek=$(($image_size / $BLOCK_SIZE))
+ # make a psuedo-random pattern block
+ dd if=/dev/urandom of=$img.fill bs=4096 count=1 2>/dev/null
+
+ # extend to 2mb
+ rm -f $img.fill.2mb
+ for x in $(seq 512); do cat $img.fill >> $img.fill.2mb; done
+
+ # extend to image size
+ local n=$(($image_size / (1048576 * 2) + 1))
+ rm -f $img
+ dd if=/dev/zero of=$img bs=1M count=1 2>/dev/null
+ for x in $(seq $n); do echo -ne "$x\r"; cat $img.fill.2mb >> $img; done
+
+ # trim to requested size
+ truncate --size=$image_size $img
+
+ # remove temporary file
+ rm -f $img.fill.2mb
+
/sbin/sfdisk -S 32 -H 32 --force -uS $img <<EOF
8192,131072,83,*
139264,,,
diff --git a/modules/sd_card_image/postimage.50.makezd.sh b/modules/sd_card_image/postimage.50.makezd.sh
index de25339..b8b17ed 100644
--- a/modules/sd_card_image/postimage.50.makezd.sh
+++ b/modules/sd_card_image/postimage.50.makezd.sh
@@ -21,6 +21,7 @@ for line in $(env); do
pfx=$outputdir/$osname
echo "Making ZD image for $osname..."
$bindir/zhashfs 0x20000 sha256 $pfx.disk.img $pfx.zsp $pfx.zd
+ rm -f $pfx.disk.img.fill
echo "Creating MD5sum of $osname.zd..."
pushd $outputdir >/dev/null
_______________________________________________
Devel mailing list
[email protected]
http://lists.laptop.org/listinfo/devel
