Re: XO-1.5 users: need your SD card data
On 26 April 2011 00:02, John Gilmore g...@toad.com wrote: There's now a mkfs option that doesn't zero all the inodes while building the file system. It instead sets an extent for the inode tables, in the superblock, and the kernel zeroes more inodes as it needs to, in parallel with ordinary system operation. It also has a similar option for the block groups. This is the lazy_itable_init option set with mke2fs -E and the uninit_bg option set with mke2fs -O. uninit_bg works only with ext4. I think lazy_itable_init works with ext3. Thanks, I had spotted this as well. I don't think its available in the kernels we ship, but should definitely be considered for the future, because it may allow us to cheaply implement automatic filesystem resizing: http://dev.laptop.org/ticket/10040 Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data
On 18 April 2011 23:46, James Cameron qu...@laptop.org wrote: No, ext* does not require unused filesystem blocks to be zeroed out or to read back 0s if read. Ok, you convinced me :) What I was getting confused with is the requirement for inode tables to be zeroed out. (this is probably what takes most of the time during mke2fs). I might be wrong on that count as well though - its just something I picked up in a discussion which I could have misunderstood. Probably easy to test: create a file of all 0xff bytes, run mke2fs on it, count the number of 0x00 bytes. So if we were simply to assume that zeroed data is unused, we might end up with problems where cards that erase to all-ones wouldn't have the required all-zeroed inode tables. But I don't think that's what you're suggesting. It seems like it might be something worth investigating further. I guess the possible complications are converting the free block numbers on the partitions into offsets into the disk itself, and then into zblock indexes. The questionable suitability for zerofree for ext4 is a small concern as well - we do use file extents. Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data
Note that Quozl's version also works better on images created with image-builder tools which delete files during the image creation process (ie, on just about anything which involves mounting the filesystem image during the image creation process; as opposed to (say) squashfs, which is a read-only filesystem created all at once from a filesystem tree at the very end). Deleted files will leave 'dirty' blocks which are not zero but can still be safely skipped. --scott -- ( http://cscott.net ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data
On Tue, Apr 19, 2011 at 04:05:38PM -0400, C. Scott Ananian wrote: Note that Quozl's version also works better on images created with image-builder tools which delete files during the image creation process ... Yes, but we don't do that. Tested on 6th August 2010 using olpc-os-builder at time, and zerofree made no significant difference. It was because root filesystem was prepared then copied into empty ext3 filesystem. No significant file operations after. Checking again now, on master of olpc-os-builder, same happens. Line 62 of modules/sd_card_image/image.50.makefs.sh is copy, with ext4 filesystem empty immediately prior. -- James Cameron http://quozl.linux.org.au/ ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data
On 15 April 2011 03:10, James Cameron qu...@laptop.org wrote: G'day Daniel, As an alternative, consider identifying the unused blocks in the filesystem, and avoid including them in the .zd file. This would make it unnecessary to know whether the bits will be set or cleared by the card. ext2, ext3, and ext4 care nothing for the bits in unused blocks. I'm not sure about that. I have been working under the assumption that ext* requires unused blocks to be zeroed out (or to read back 0s if read), which seems to be in agreement with the rest of your mail. However, if we use a SD card that reads back all-ones for unused blocks, I think we would run into problems. Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data
On 14 April 2011 10:26, Daniel Drake d...@laptop.org wrote: I may have found a way to make fs-update run on steroids. But I need to know if all/most SD cards are like mine. Thanks everyone, I think thats enough data for now. If the reported information is correct, the optimization I have found would apply to all cards reported except for Kevin Gordon's 8gb one. Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data
On Mon, Apr 18, 2011 at 04:02:08PM +0100, Daniel Drake wrote:
On 15 April 2011 03:10, James Cameron qu...@laptop.org wrote:
G'day Daniel,
As an alternative, consider identifying the unused blocks in the
filesystem, and avoid including them in the .zd file. ?This would make
it unnecessary to know whether the bits will be set or cleared by the
card. ?ext2, ext3, and ext4 care nothing for the bits in unused blocks.
I'm not sure about that. I have been working under the assumption that
ext* requires unused blocks to be zeroed out (or to read back 0s if
read), which seems to be in agreement with the rest of your mail.
No, ext* does not require unused filesystem blocks to be zeroed out or
to read back 0s if read.
I don't see how you thought that would be in agreement with the rest of
my mail, which went on to describe how zerofree can be used to identify
unused filesystem blocks. Once identified they may be omitted from the
.zd file.
mke2fs does not write zeroes to all unused filesystem blocks. I've
tested this and the results are below.
A file deletion that causes filesystem blocks to become unused does not
write zeroes to the blocks. While deleting a large file on ext3 can
take a while, it is not because it is zeroing the blocks.
However, if we use a SD card that reads back all-ones for unused
blocks, I think we would run into problems.
Assuming you are dealing with unused filesystem blocks ... I'm certain
you would not run into problems with an SD card that reads back all-ones
for erased blocks. If you are running into problems, then perhaps they
relate to the failure of Q3A64 to properly erase the SD card during
fs-update, which I covered in another thread and since fixed.
As confirmation that reading back all-ones for unused filesystem blocks
is harmless, I've made a variant of zerofree that changes all the unused
filesystem blocks to 0xff, and the laptop operated normally after
reboot. The patch for this is attached as zerofree-fill-0xff.patch
As confirmation that mke2fs does not write zeroes to all unused
filesystem blocks, a test method is to make a file containing non-zero
but identifiable data, make a filesystem over the file, count the number
of blocks that have been changed, and then use zerofree to demonstrate
that there are non-zero free blocks:
# create a psuedo-random 512-byte block
cd /tmp
dd if=/dev/urandom bs=512 count=1 of=key
# replicate it to 1MB size
rm -f key.1mb
for x in $(seq 2048); do cat key key.1mb; done
# replicate it to 4GB size
MB=4096
rm -f img
for x in $(seq ${MB}); do echo -ne $x\r; cat key.1mb img; done
# verify that all image blocks contain the original pattern
# source:
# http://dev.laptop.org/~quozl/count-changed-blocks.c (download)
# http://dev.laptop.org/~quozl/count-changed-blocks.c.html (view)
count-changed-blocks key img
# result: zero blocks changed from original pattern
# create an ext3 filesystem on the image,
# which will write filesystem metadata.
mke2fs -F -j img
# count the blocks changed by mke2fs
count-changed-blocks key img mke2fs-changed-blocks
# result: 396294 times 512-byte blocks
# (equivalent to about 96 filesystem blocks, or 193MB)
# count and zero the free filesystem blocks
zerofree -v img
# result: 997249/997265/1048576 nonzero/free/total 4096-byte blocks
If mke2fs did write zeroes to unused filesystem blocks, then (a) the
number of changed image blocks would have approached the size of the
image, and (b) zerofree would have reported no filesystem blocks
nonzero.
olpc-os-builder does start with an image file that is derived from
/dev/zero, per modules/sd_card_image/image.50.makefs.sh ... therefore
unused filesystem blocks will typically be zeroed in the .zd file, but
the converse is not true: a zeroed image block in the .zd file won't
necessarily be an unused filesystem block.
What I'm proposing is that unused filesystem blocks not be embedded in
the .zd file, and as a result fs-update will erase the corresponding SD
block but not write to it. Since writing to the SD card is the most
expensive operation, time will be saved, and it won't be necessary to
know what the card sets the bits to on erase.
(For reference, on the Toshiba SD parts I've been testing, an
erase-blocks operation is very fast, and apparently only changes
metadata in the card's flash translation layer ... but doesn't actually
perform flash level erasure. Physical erasure still has to occur during
write. The fs-update elapsed time is unchanged.)
--
James Cameron
http://quozl.linux.org.au/
Only in /c/tmp/olpc/zerofree-1.0.1: zerofree
diff -ur zerofree-1.0.1/zerofree.c /c/tmp/olpc/zerofree-1.0.1/zerofree.c
--- zerofree-1.0.1/zerofree.c 2007-08-12 19:38:31.0 +1000
+++ /c/tmp/olpc/zerofree-1.0.1/zerofree.c 2011-04-19 08:41:53.0 +1000
@@ -78,6 +78,7 @@
}
empty = (unsigned char *)calloc(1, current_fs-blocksize) ;
+ memset(empty, 0xff, current_fs-blocksize);
buf = (unsigned char *)malloc(current_fs-blocksize) ;
Re: XO-1.5 users: need your SD card data
On 15 April 2011 02:26, Daniel Drake d...@laptop.org wrote: Hi, I may have found a way to make fs-update run on steroids. But I need to know if all/most SD cards are like mine. Please open your XO-1.5 and run: cat /sys/devices/pci:00/:00:0c.0/mmc_host/mmc2/mmc2:*/scr And post the output here. Please state clearly if this is a SD card that you installed yourself, or if its the one that was shipped in the laptop by default from OLPC. I got the same on two XO-1.5s: 002358000 No changes from the factory: SKU 104. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data
On Thu, Apr 14, 2011 at 10:10 PM, James Cameron qu...@laptop.org wrote: As an alternative, consider identifying the unused blocks in the filesystem, and avoid including them in the .zd file. This would make it unnecessary to know whether the bits will be set or cleared by the card. ext2, ext3, and ext4 care nothing for the bits in unused blocks. This would also nicely finesse the problem that different cards may erase to different values (0xFF vs 0x00). --scott -- ( http://cscott.net ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
XO-1.5 users: need your SD card data
Hi, I may have found a way to make fs-update run on steroids. But I need to know if all/most SD cards are like mine. Please open your XO-1.5 and run: cat /sys/devices/pci:00/:00:0c.0/mmc_host/mmc2/mmc2:*/scr And post the output here. Please state clearly if this is a SD card that you installed yourself, or if its the one that was shipped in the laptop by default from OLPC. The optimization I'm looking at is that fs-update first erases the disk (which only takes a split second) and then spends most of its time writing zeroes, but for some SD cards (hopefully most of them), erasing the disk zeroes out the whole thing anyway. So no need to spend loads of time writing zeroes. Thanks, Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data
On Thu, Apr 14, 2011 at 5:26 PM, Daniel Drake d...@laptop.org wrote: Hi, I may have found a way to make fs-update run on steroids. But I need to know if all/most SD cards are like mine. Please open your XO-1.5 and run: cat /sys/devices/pci:00/:00:0c.0/mmc_host/mmc2/mmc2:*/scr And post the output here. Please state clearly if this is a SD card that you installed yourself, or if its the one that was shipped in the laptop by default from OLPC. The optimization I'm looking at is that fs-update first erases the disk (which only takes a split second) and then spends most of its time writing zeroes, but for some SD cards (hopefully most of them), erasing the disk zeroes out the whole thing anyway. So no need to spend loads of time writing zeroes. Do you want that sort of data just for the SD cards used specifically in the XO. I have around 20 cards used for various linux hosts and I can give you the info for them all if its useful. Also generally writing zeros to any flash memory that implements its own wear levelling algorithms I would have though wouldn't be of much use. Peter ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data
original from olpc (the same value in 3 XOs) 02358000 Gonzalo On Thu, Apr 14, 2011 at 4:26 PM, Daniel Drake d...@laptop.org wrote: Hi, I may have found a way to make fs-update run on steroids. But I need to know if all/most SD cards are like mine. Please open your XO-1.5 and run: cat /sys/devices/pci:00/:00:0c.0/mmc_host/mmc2/mmc2:*/scr And post the output here. Please state clearly if this is a SD card that you installed yourself, or if its the one that was shipped in the laptop by default from OLPC. The optimization I'm looking at is that fs-update first erases the disk (which only takes a split second) and then spends most of its time writing zeroes, but for some SD cards (hopefully most of them), erasing the disk zeroes out the whole thing anyway. So no need to spend loads of time writing zeroes. Thanks, Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data [Devel Digest, Vol 62, Issue 28]
original from olpc (the same value in 3 XOs) 02358000 Same on 2 XO-1.5. One C2 one C3 ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: XO-1.5 users: need your SD card data
G'day Daniel,
As an alternative, consider identifying the unused blocks in the
filesystem, and avoid including them in the .zd file. This would make
it unnecessary to know whether the bits will be set or cleared by the
card. ext2, ext3, and ext4 care nothing for the bits in unused blocks.
zerofree [1] demonstrates how to identify the unused blocks. The
attached patch to zerofree-1.0.1 lets me dump the block numbers of the
unused blocks without changing the filesystem:
zerofree -nd filesystem.img filesystem-free-blocks
The block numbers might then be used by a modified zhashfs, or zhashfs
might make the same block bitmap test calls that zerofree makes.
It is said by the author [1] and in various places that zerofree can
work with ext4 ... I've run it against the ext4 partition of an SD card
that contains os14 ... with no apparent ill-effects.
However, Richard Jones suggests [2] that files with extents may not be
handled. I'm not sure if that is a concern for olpc-os-builder images.
References:
1. http://intgat.tigress.co.uk/rmy/uml/
2. http://www.redhat.com/archives/libguestfs/2010-June/msg00019.html
--
James Cameron
http://quozl.linux.org.au/
--- zerofree.c.orig 2011-04-15 11:40:12.0 +1000
+++ zerofree.c 2011-04-15 11:40:44.0 +1000
@@ -36,8 +36,9 @@
int old_percent ;
int verbose = 0 ;
int dryrun = 0 ;
+ int dump = 0 ;
- while ( (c=getopt(argc, argv, nv)) != -1 ) {
+ while ( (c=getopt(argc, argv, nvd)) != -1 ) {
switch (c) {
case 'n' :
dryrun = 1 ;
@@ -45,6 +46,9 @@
case 'v' :
verbose = 1 ;
break ;
+ case 'd' :
+ dump = 1 ;
+ break ;
default :
fprintf(stderr, USAGE, argv[0]) ;
return 1 ;
@@ -114,6 +118,10 @@
++free ;
+ if ( dump ) {
+ fprintf(stderr, %ld\n, blk);
+ }
+
percent = 100.0 * (double)free/
(double)current_fs-super-s_free_blocks_count ;
___
Devel mailing list
Devel@lists.laptop.org
http://lists.laptop.org/listinfo/devel
