cramfs filesystem patch for 2.4.0
Here is an update of my patch for the cramfs filesystem for 2.4.0 with a
few minor additions. See below for the list of changes.
It's backward-compatible and only modifies cramfs code (aside from
adding cramfs to struct super_block). All old cramfs images will still
work and new cramfs images are mountable on old kernels if you avoid
using new features: holes (holes require kernel 2.3.39 or later) and
padding for a boot sector (you should have a new kernel in that case).
Most of this code has been fairly well-tested, but a few features are
newer and less well-tested (like the directory sorting). If you use
cramfs, please try it out and let me know if you have any feedback,
comments, or patches. (I've started looking at adding some of the
existing patches, although I'll need help testing most of them.)
Dan
15 Jan 2001:
- update for 2.4.0
- move fs/cramfs/cramfs.h to include/linux/cramfs_fs.h
- add magic to in-memory version of cramfs superblock (for initrd support)
- renamed a few #defines in cramfs_fs.h
07 Dec 2000:
superblock changes:
- revised fsid field (still unique, contains useful fs info:
a CRC, edition number for implementation-specific versioning,
block and file counts for statfs)
- size field is now used
- new feature flags (fsid version 2, holes, shifted root for boot loader,
directory sorting)
inode.c
- proper checking for device size (torvalds)
- speed up IO a bit - read in the block asynchronously rather than
using bread() on them one by one (torvalds)
- early directory lookup exit (since directories are now sorted)
- add reporting of blocks and blksize for statfs
- removed superfluous test for cramfs signature
cramfsck.c
- new program to fsck/extract cramfs images
mkcramfs.c
- CRC added to filesystem image
- allow holes to be created (see -z option)
- support for a cramfs boot loader (see -p option)
- support for inserting an kernel image into a cramfs image (see -i option)
- directory sorting (more consistent images, allows faster lookups)
- bug fix: allocate enough RAM for the fs image (small images)
- bug fix: work around a bug in ramfs where it would
incorrectly report the number of blocks for file (a problem when
creating cramfs images from a ramfs filesystem)
- options to set name/edition number in superblock (-n and -e, respectively)
cramfs.txt:
- added entry for /etc/magic to cramfs.txt documentation
cleanup:
- moved width definitions to cramfs.h
- define special cramfs super block
diff -urN linux-2.4.0-orig/Documentation/filesystems/cramfs.txt
linux/Documentation/filesystems/cramfs.txt
--- linux-2.4.0-orig/Documentation/filesystems/cramfs.txt Sat May 20 11:30:31
2000
+++ linux/Documentation/filesystems/cramfs.txt Sat Jan 13 18:50:47 2001
@@ -47,6 +47,21 @@
mind the filesystem becoming unreadable to future kernels.
+For /usr/share/magic
+--
+
+0 long0x28cd3d45 Linux cramfs
+>4 longx size %d
+>8 longx flags 0x%x
+>12longx future 0x%x
+>16string >\0 signature "%.16s"
+>32longx fsid.crc 0x%x
+>36longx fsid.edition %d
+>40longx fsid.blocks %d
+>44longx fsid.files %d
+>48string >\0 name "%.16s"
+
+
Hacker Notes
diff -urN linux-2.4.0-orig/fs/cramfs/README linux/fs/cramfs/README
--- linux-2.4.0-orig/fs/cramfs/README Tue Jan 11 10:24:58 2000
+++ linux/fs/cramfs/README Sat Jan 13 18:50:47 2001
@@ -14,11 +14,11 @@
-: struct cramfs_super (see cramfs.h).
+: struct cramfs_super (see cramfs_fs.h).
:
For each file:
- struct cramfs_inode (see cramfs.h).
+ struct cramfs_inode (see cramfs_fs.h).
Filename. Not generally null-terminated, but it is
null-padded to a multiple of 4 bytes.
diff -urN linux-2.4.0-orig/fs/cramfs/cramfs.h linux/fs/cramfs/cramfs.h
--- linux-2.4.0-orig/fs/cramfs/cramfs.h Tue Jan 11 10:24:58 2000
+++ linux/fs/cramfs/cramfs.hWed Dec 31 16:00:00 1969
@@ -1,52 +0,0 @@
-#ifndef __CRAMFS_H
-#define __CRAMFS_H
-
-#define CRAMFS_MAGIC 0x28cd3d45 /* some random number */
-#define CRAMFS_SIGNATURE "Compressed ROMFS"
-
-/*
- * Reasonably terse representation of the inode data.
- */
-struct cramfs_inode {
- u32 mode:16, uid:16;
- /* SIZE for device files is i_rdev */
- u32 size:24, gid:8;
- /* NAMELEN is the length of the file name, divided by 4 and
- rounded up. (cramfs doesn't support hard links.) */
- /* OFFSET: For symlinks and non-empty regular files, this
- contains the offset (divided by 4) of the file data in
- compressed
cramfs filesystem patch for 2.4.0
Here is an update of my patch for the cramfs filesystem for 2.4.0 with a
few minor additions. See below for the list of changes.
It's backward-compatible and only modifies cramfs code (aside from
adding cramfs to struct super_block). All old cramfs images will still
work and new cramfs images are mountable on old kernels if you avoid
using new features: holes (holes require kernel 2.3.39 or later) and
padding for a boot sector (you should have a new kernel in that case).
Most of this code has been fairly well-tested, but a few features are
newer and less well-tested (like the directory sorting). If you use
cramfs, please try it out and let me know if you have any feedback,
comments, or patches. (I've started looking at adding some of the
existing patches, although I'll need help testing most of them.)
Dan
15 Jan 2001:
- update for 2.4.0
- move fs/cramfs/cramfs.h to include/linux/cramfs_fs.h
- add magic to in-memory version of cramfs superblock (for initrd support)
- renamed a few #defines in cramfs_fs.h
07 Dec 2000:
superblock changes:
- revised fsid field (still unique, contains useful fs info:
a CRC, edition number for implementation-specific versioning,
block and file counts for statfs)
- size field is now used
- new feature flags (fsid version 2, holes, shifted root for boot loader,
directory sorting)
inode.c
- proper checking for device size (torvalds)
- speed up IO a bit - read in the block asynchronously rather than
using bread() on them one by one (torvalds)
- early directory lookup exit (since directories are now sorted)
- add reporting of blocks and blksize for statfs
- removed superfluous test for cramfs signature
cramfsck.c
- new program to fsck/extract cramfs images
mkcramfs.c
- CRC added to filesystem image
- allow holes to be created (see -z option)
- support for a cramfs boot loader (see -p option)
- support for inserting an kernel image into a cramfs image (see -i option)
- directory sorting (more consistent images, allows faster lookups)
- bug fix: allocate enough RAM for the fs image (small images)
- bug fix: work around a bug in ramfs where it would
incorrectly report the number of blocks for file (a problem when
creating cramfs images from a ramfs filesystem)
- options to set name/edition number in superblock (-n and -e, respectively)
cramfs.txt:
- added entry for /etc/magic to cramfs.txt documentation
cleanup:
- moved width definitions to cramfs.h
- define special cramfs super block
diff -urN linux-2.4.0-orig/Documentation/filesystems/cramfs.txt
linux/Documentation/filesystems/cramfs.txt
--- linux-2.4.0-orig/Documentation/filesystems/cramfs.txt Sat May 20 11:30:31
2000
+++ linux/Documentation/filesystems/cramfs.txt Sat Jan 13 18:50:47 2001
@@ -47,6 +47,21 @@
mind the filesystem becoming unreadable to future kernels.
+For /usr/share/magic
+--
+
+0 long0x28cd3d45 Linux cramfs
+4 longx size %d
+8 longx flags 0x%x
+12longx future 0x%x
+16string \0 signature "%.16s"
+32longx fsid.crc 0x%x
+36longx fsid.edition %d
+40longx fsid.blocks %d
+44longx fsid.files %d
+48string \0 name "%.16s"
+
+
Hacker Notes
diff -urN linux-2.4.0-orig/fs/cramfs/README linux/fs/cramfs/README
--- linux-2.4.0-orig/fs/cramfs/README Tue Jan 11 10:24:58 2000
+++ linux/fs/cramfs/README Sat Jan 13 18:50:47 2001
@@ -14,11 +14,11 @@
directory_structure
data
-superblock: struct cramfs_super (see cramfs.h).
+superblock: struct cramfs_super (see cramfs_fs.h).
directory_structure:
For each file:
- struct cramfs_inode (see cramfs.h).
+ struct cramfs_inode (see cramfs_fs.h).
Filename. Not generally null-terminated, but it is
null-padded to a multiple of 4 bytes.
diff -urN linux-2.4.0-orig/fs/cramfs/cramfs.h linux/fs/cramfs/cramfs.h
--- linux-2.4.0-orig/fs/cramfs/cramfs.h Tue Jan 11 10:24:58 2000
+++ linux/fs/cramfs/cramfs.hWed Dec 31 16:00:00 1969
@@ -1,52 +0,0 @@
-#ifndef __CRAMFS_H
-#define __CRAMFS_H
-
-#define CRAMFS_MAGIC 0x28cd3d45 /* some random number */
-#define CRAMFS_SIGNATURE "Compressed ROMFS"
-
-/*
- * Reasonably terse representation of the inode data.
- */
-struct cramfs_inode {
- u32 mode:16, uid:16;
- /* SIZE for device files is i_rdev */
- u32 size:24, gid:8;
- /* NAMELEN is the length of the file name, divided by 4 and
- rounded up. (cramfs doesn't support hard links.) */
- /* OFFSET: For symlinks and non-empty regular files, this
- contains the offset
Re: cramfs filesystem patch
On Saturday 09 December 2000 06:39, Tim Riker wrote:
> I'd like to see these patches as well. They may be useful on the iPAQ
> (and similar hardware like my Yopy here... ;-)
>
> I wish some hardware vendor out there would build an x86 box that used
> memory addressable flash from 0 up and RAM up higher. A simple Linux
> kernel boot loader could then replace the BIOS. Flash would be less
> expensive than 8086 lower memory windowed flash hacks.
>
> Has any work been done on e2compr in 2.4 yet?
>
Patch inserted. It's not ready for kernel inclusion or anything..., there are some
remaining issues to fix that I haven't figured out good ways to handle yet.
For instance, we still use a /dev/rom to bootstrap the kernel as to which
file system module to pull up. Then all the read/writes completely by pass
/dev/rom. Silly, I know, but I haven't figured out a smart way to deal with
this particular problem.
Sorry for the late reply..., your message got lost in a sea of mail. You might
find some other interesting stuff on the ftp.agendacomputing.com ftp site.
XIP MTD flash read/write, which I haven't forward ported to MTDs CVS yet.
(Working on it) But that would allow if you guys can get XIP kernel working
in ARM like we have in MIPS to read and write to flash. Other randomn stuff,
and things get added with a slightly normal frequency. One benefit with this
patch is copying, the other is dumping that 32k buffer, and giving it back to
pageable memory.
Thanks,
Shane.
diff -urN linux.orig/fs/Config.in linux/fs/Config.in
--- linux.orig/fs/Config.in Fri Oct 27 04:23:18 2000
+++ linux/fs/Config.in Fri Oct 27 03:57:52 2000
@@ -29,6 +29,10 @@
int 'JFFS debugging verbosity (0 = quiet, 3 = noisy)' CONFIG_JFFS_FS_VERBOSE 0
fi
tristate 'Compressed ROM file system support' CONFIG_CRAMFS
+dep_mbool 'Linear addressing for CRAMFS' CONFIG_CRAMFS_LINEAR_ADDRESSING
+$CONFIG_CRAMFS
+if [ "$CONFIG_CRAMFS_LINEAR_ADDRESSING" != "n" ] ; then
+ hex 'Starting address for CRAMFS filesystem' CONFIG_CRAMFS_LA_ADDRESS bf28
+fi
tristate 'Simple RAM-based file system support' CONFIG_RAMFS
tristate 'ISO 9660 CDROM file system support' CONFIG_ISO9660_FS
diff -urN linux.orig/fs/cramfs/inode.c linux/fs/cramfs/inode.c
--- linux.orig/fs/cramfs/inode.cFri Oct 27 04:22:36 2000
+++ linux/fs/cramfs/inode.c Fri Oct 27 04:30:18 2000
@@ -11,6 +11,20 @@
* The actual compression is based on zlib, see the other files.
*/
+/* Linear Addressing code
+ *
+ * Copyright (C) 2000 Shane Nay.
+ *
+ * Allows you to have a linearly addressed cramfs filesystem.
+ * Saves the need for buffer, and the munging of the buffer.
+ * Savings a bit over 32k with default PAGE_SIZE, BUFFER_SIZE
+ * etc. Usefull on embedded platform with ROM :-).
+ *
+ * Downsides- Currently linear addressed cramfs partitions
+ * don't co-exist with block cramfs partitions.
+ *
+ */
+
#include
#include
#include
@@ -68,6 +82,23 @@
return inode;
}
+#if defined(CONFIG_CRAMFS_CRAMFS_LINEAR_ADDRESSING) && defined
+(CONFIG_CRAMFS_LA_ADDRESS)
+
+/*
+ * Returns a pointer to the linearly addressed filesystem.
+ * Simple byte size pointer addition.
+ */
+static unsigned char* romdisk_top=(unsigned char*) CONFIG_CRAMFS_LA_ADDRESS;
+
+static void *cramfs_read(struct super_block *sb, unsigned int offset, unsigned int
+len)
+{
+ if (!len)
+ return NULL;
+ return romdisk_top + offset;
+}
+
+#else /* !CONFIG_CRAMFS_LINEAR_ADDRESSING aka Regular block mode */
+
/*
* We have our own block cache: don't fill up the buffer cache
* with the rom-image, because the way the filesystem is set
@@ -149,6 +180,8 @@
}
return read_buffers[buffer] + offset;
}
+
+#endif
static struct super_block * cramfs_read_super(struct super_block *sb, void *data, int
silent)
@@ -161,10 +194,11 @@
set_blocksize(sb->s_dev, PAGE_CACHE_SIZE);
sb->s_blocksize = PAGE_CACHE_SIZE;
sb->s_blocksize_bits = PAGE_CACHE_SHIFT;
+#if !( defined(CONFIG_CRAMFS_CRAMFS_LINEAR_ADDRESSING) && defined
+(CONFIG_CRAMFS_LA_ADDRESS) )
/* Invalidate the read buffers on mount: think disk change.. */
for (i = 0; i < READ_BUFFERS; i++)
buffer_blocknr[i] = -1;
+#endif
/* Read the first block and get the superblock from it */
memcpy(, cramfs_read(sb, 0, sizeof(super)), sizeof(super));
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
Re: cramfs filesystem patch
On Saturday 09 December 2000 06:39, Tim Riker wrote:
I'd like to see these patches as well. They may be useful on the iPAQ
(and similar hardware like my Yopy here... ;-)
I wish some hardware vendor out there would build an x86 box that used
memory addressable flash from 0 up and RAM up higher. A simple Linux
kernel boot loader could then replace the BIOS. Flash would be less
expensive than 8086 lower memory windowed flash hacks.
Has any work been done on e2compr in 2.4 yet?
Patch inserted. It's not ready for kernel inclusion or anything..., there are some
remaining issues to fix that I haven't figured out good ways to handle yet.
For instance, we still use a /dev/rom to bootstrap the kernel as to which
file system module to pull up. Then all the read/writes completely by pass
/dev/rom. Silly, I know, but I haven't figured out a smart way to deal with
this particular problem.
Sorry for the late reply..., your message got lost in a sea of mail. You might
find some other interesting stuff on the ftp.agendacomputing.com ftp site.
XIP MTD flash read/write, which I haven't forward ported to MTDs CVS yet.
(Working on it) But that would allow if you guys can get XIP kernel working
in ARM like we have in MIPS to read and write to flash. Other randomn stuff,
and things get added with a slightly normal frequency. One benefit with this
patch is copying, the other is dumping that 32k buffer, and giving it back to
pageable memory.
Thanks,
Shane.
diff -urN linux.orig/fs/Config.in linux/fs/Config.in
--- linux.orig/fs/Config.in Fri Oct 27 04:23:18 2000
+++ linux/fs/Config.in Fri Oct 27 03:57:52 2000
@@ -29,6 +29,10 @@
int 'JFFS debugging verbosity (0 = quiet, 3 = noisy)' CONFIG_JFFS_FS_VERBOSE 0
fi
tristate 'Compressed ROM file system support' CONFIG_CRAMFS
+dep_mbool 'Linear addressing for CRAMFS' CONFIG_CRAMFS_LINEAR_ADDRESSING
+$CONFIG_CRAMFS
+if [ "$CONFIG_CRAMFS_LINEAR_ADDRESSING" != "n" ] ; then
+ hex 'Starting address for CRAMFS filesystem' CONFIG_CRAMFS_LA_ADDRESS bf28
+fi
tristate 'Simple RAM-based file system support' CONFIG_RAMFS
tristate 'ISO 9660 CDROM file system support' CONFIG_ISO9660_FS
diff -urN linux.orig/fs/cramfs/inode.c linux/fs/cramfs/inode.c
--- linux.orig/fs/cramfs/inode.cFri Oct 27 04:22:36 2000
+++ linux/fs/cramfs/inode.c Fri Oct 27 04:30:18 2000
@@ -11,6 +11,20 @@
* The actual compression is based on zlib, see the other files.
*/
+/* Linear Addressing code
+ *
+ * Copyright (C) 2000 Shane Nay.
+ *
+ * Allows you to have a linearly addressed cramfs filesystem.
+ * Saves the need for buffer, and the munging of the buffer.
+ * Savings a bit over 32k with default PAGE_SIZE, BUFFER_SIZE
+ * etc. Usefull on embedded platform with ROM :-).
+ *
+ * Downsides- Currently linear addressed cramfs partitions
+ * don't co-exist with block cramfs partitions.
+ *
+ */
+
#include linux/module.h
#include linux/fs.h
#include linux/pagemap.h
@@ -68,6 +82,23 @@
return inode;
}
+#if defined(CONFIG_CRAMFS_CRAMFS_LINEAR_ADDRESSING) defined
+(CONFIG_CRAMFS_LA_ADDRESS)
+
+/*
+ * Returns a pointer to the linearly addressed filesystem.
+ * Simple byte size pointer addition.
+ */
+static unsigned char* romdisk_top=(unsigned char*) CONFIG_CRAMFS_LA_ADDRESS;
+
+static void *cramfs_read(struct super_block *sb, unsigned int offset, unsigned int
+len)
+{
+ if (!len)
+ return NULL;
+ return romdisk_top + offset;
+}
+
+#else /* !CONFIG_CRAMFS_LINEAR_ADDRESSING aka Regular block mode */
+
/*
* We have our own block cache: don't fill up the buffer cache
* with the rom-image, because the way the filesystem is set
@@ -149,6 +180,8 @@
}
return read_buffers[buffer] + offset;
}
+
+#endif
static struct super_block * cramfs_read_super(struct super_block *sb, void *data, int
silent)
@@ -161,10 +194,11 @@
set_blocksize(sb-s_dev, PAGE_CACHE_SIZE);
sb-s_blocksize = PAGE_CACHE_SIZE;
sb-s_blocksize_bits = PAGE_CACHE_SHIFT;
+#if !( defined(CONFIG_CRAMFS_CRAMFS_LINEAR_ADDRESSING) defined
+(CONFIG_CRAMFS_LA_ADDRESS) )
/* Invalidate the read buffers on mount: think disk change.. */
for (i = 0; i READ_BUFFERS; i++)
buffer_blocknr[i] = -1;
+#endif
/* Read the first block and get the superblock from it */
memcpy(super, cramfs_read(sb, 0, sizeof(super)), sizeof(super));
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
Re: cramfs filesystem patch
Daniel, > Have you done a comparison of LZO against zlib (decompression > speed/size vs. compression ratio)? It uses less RAM/CPU to decompress > at the cost of wasting storage space, but it's hard to make a decision > without real numbers. I can't do a test on speed because I haven't had time to get it to work in the kernel. But it does take no memory during decompression. The resulting size of the romdisk is increased by 5% over zlib's lightest compression. So it does create a larger cramfs partition, but the lower (read none) memory overhead is a big plus, and decompression speed is a big plus. > Actually, my most major concern is that LZO may not be legally > unencumbered. There's no mention of any patent search in the LZO > site. zlib has been very well researched in this regard. Yes..., this is my major concern as well. It's why I sort of opted not to pour tons more time on it when I was working on it. (I was having a byte alignment problem) > Do you want to implement more than one type of compression per cramfs > filesystem or are the compression flags in the inode only needed to > distinguish between uncompressed (XIP) and compressed data? I assume > you wouldn't use both LZO and zlib code, except perhaps for backward > compatibility with your own products. > > If you're using XIP and uncompressed binaries, you may want to think > about using romfs for them. Yes, that's where it's being implemented right now is in romfs. But I want to cross it over into cramfs to be able to support more than one type of file compression type..., one being zlib, the other being not compressed and aligned for XIP usage. > Also regarding leaving binaries/libraries uncompressed, they tend to > be about 50% of a small (uncompressed) filesystem in my experience. > Is the ratio much different in the Agenda? More like 72% :-). But we're not planning on having all the libraries and executables running XIP, just a few key ones..., X, Xlibs, etc. That's why integration with cramfs is sort of key, one other reason too being that we want their to be only one flash to put all our s/w on. If we pull them out as seperate pieces, then we've got three seperate flashes for our users to deal with when they do a full upgrade. (Kernel, cramfs, romfs) We'd like to avoid that if possible. > I want to be careful about increasing the size of the inode. > Increasing it a small amount may have some benefits, but I've been > trying really hard to avoid it. I have been considering a version 2 > inode that would change the allocation of bits in the inode (to allow > larger files at the expense of uid/gid space). > > I think it would be better to put the type of compression at the > beginning of each data block. zlib has flags for this sort of thing > and there's already a header on each zlib data block in cramfs. Hmm..., maybe. But that's a little hacky probably..., I don't know, it seems like that info should go into the inode to me. I suppose we could pull it out and put it in a header, or make our own version of cramfs that does less gids/uids and use some of those bits as flags for filetypes. But I'd rather not fork things up like that. > BTW, how many uids/gids do you need cramfs to support? My belief is > that 16 (or 256 at most) uids/gids is more than enough embedded system. In the cramfs partition..., we probably only need 4 really. The users can create their own accounts, etc., but they exist on another partition type (JFFS) which can support as many gids, uids, etc. as ext2fs. Thanks, Shane. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: cramfs filesystem patch
Daniel, Have you done a comparison of LZO against zlib (decompression speed/size vs. compression ratio)? It uses less RAM/CPU to decompress at the cost of wasting storage space, but it's hard to make a decision without real numbers. I can't do a test on speed because I haven't had time to get it to work in the kernel. But it does take no memory during decompression. The resulting size of the romdisk is increased by 5% over zlib's lightest compression. So it does create a larger cramfs partition, but the lower (read none) memory overhead is a big plus, and decompression speed is a big plus. Actually, my most major concern is that LZO may not be legally unencumbered. There's no mention of any patent search in the LZO site. zlib has been very well researched in this regard. Yes..., this is my major concern as well. It's why I sort of opted not to pour tons more time on it when I was working on it. (I was having a byte alignment problem) Do you want to implement more than one type of compression per cramfs filesystem or are the compression flags in the inode only needed to distinguish between uncompressed (XIP) and compressed data? I assume you wouldn't use both LZO and zlib code, except perhaps for backward compatibility with your own products. If you're using XIP and uncompressed binaries, you may want to think about using romfs for them. Yes, that's where it's being implemented right now is in romfs. But I want to cross it over into cramfs to be able to support more than one type of file compression type..., one being zlib, the other being not compressed and aligned for XIP usage. Also regarding leaving binaries/libraries uncompressed, they tend to be about 50% of a small (uncompressed) filesystem in my experience. Is the ratio much different in the Agenda? More like 72% :-). But we're not planning on having all the libraries and executables running XIP, just a few key ones..., X, Xlibs, etc. That's why integration with cramfs is sort of key, one other reason too being that we want their to be only one flash to put all our s/w on. If we pull them out as seperate pieces, then we've got three seperate flashes for our users to deal with when they do a full upgrade. (Kernel, cramfs, romfs) We'd like to avoid that if possible. I want to be careful about increasing the size of the inode. Increasing it a small amount may have some benefits, but I've been trying really hard to avoid it. I have been considering a version 2 inode that would change the allocation of bits in the inode (to allow larger files at the expense of uid/gid space). I think it would be better to put the type of compression at the beginning of each data block. zlib has flags for this sort of thing and there's already a header on each zlib data block in cramfs. Hmm..., maybe. But that's a little hacky probably..., I don't know, it seems like that info should go into the inode to me. I suppose we could pull it out and put it in a header, or make our own version of cramfs that does less gids/uids and use some of those bits as flags for filetypes. But I'd rather not fork things up like that. BTW, how many uids/gids do you need cramfs to support? My belief is that 16 (or 256 at most) uids/gids is more than enough embedded system. In the cramfs partition..., we probably only need 4 really. The users can create their own accounts, etc., but they exist on another partition type (JFFS) which can support as many gids, uids, etc. as ext2fs. Thanks, Shane. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: cramfs filesystem patch
I'd like to see these patches as well. They may be useful on the iPAQ (and similar hardware like my Yopy here... ;-) I wish some hardware vendor out there would build an x86 box that used memory addressable flash from 0 up and RAM up higher. A simple Linux kernel boot loader could then replace the BIOS. Flash would be less expensive than 8086 lower memory windowed flash hacks. Has any work been done on e2compr in 2.4 yet? Shane Nay wrote: > > On Friday 08 December 2000 05:11, Daniel Quinlan wrote: > > Here's a patch for the cramfs filesystem. Lots of improvements and a > > new cramfsck program, see below for the full list of changes. > > > > It only modifies cramfs code (aside from adding cramfs to struct > > super_block) and aims to be completely backward-compatible. All old > > cramfs images will still work and new cramfs images are mountable on > > old kernels if you avoid using holes (requires 2.3.39 or later) and > > you don't pad the beginning of the cramfs filesystem for a boot sector > > (and presumably, you have a new kernel in that case). > > > > Most of this code has been fairly well-tested, but a few features are > > newer and less well-tested (like the directory sorting). > > > > If you use cramfs, please try it out and let me know if you have any > > feedback or comments (please Cc me if you respond here). > > > > Dan > > > > I've read the patch, and the added benefits of sorting seem usefull. I have > some general questions with regard to direction for Cramfs while the subject > is broached however. > > On the Agenda we use cramfs as our storage mechanism, with some linear > addressing patches since it's on flash, and the memory copies inherent with a > block device are unnecesary in our case. (And we kill the buffer of > compressed data in cramfs, as it's unnecessary as well) Going forward though > we're working on some other modifications of the the cramfs filesystem that > allow it todo some other stuff that requires an expansion of the inode, Linus > mentions in the README for cramfs that it may be usefull to expand it, but I > wanted to get peoples reactions on expanding it by 32bits, and using a part > of that to denote file information like type of compression, etc. > > I had worked previously on hacking in LZO compression into cramfs, but hadn't > had time to finish when called onto other things. The other thing we're > working on with ROMDISK is putting in XIP mode for binaries/libs that we're > going to leave uncompressed, and aligned so they don't need to be copied to > memory before executing. I'd like to integrate XIP into cramfs at some point > to allow the flash releases from agenda to be released as a single unit, but > doing so necesitates an expansion of the inode to denote this information > about certain executables/libs. So what do you all think of expanding the > inode for this purpose, and is loss of backward compatibility a massivly evil > thing, so when I make this patch I need to keep information based on the > magic of the cramfs filesystem to denote versioning information to allow for > the difference in the inode. > > Thanks, > Shane. > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > Please read the FAQ at http://www.tux.org/lkml/ -- Tim Riker - http://rikers.org/ - short SIGs! All I need to know I could have learned in Kindergarten ... if I'd just been paying attention. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: cramfs filesystem patch
On Friday 08 December 2000 05:11, Daniel Quinlan wrote: > Here's a patch for the cramfs filesystem. Lots of improvements and a > new cramfsck program, see below for the full list of changes. > > It only modifies cramfs code (aside from adding cramfs to struct > super_block) and aims to be completely backward-compatible. All old > cramfs images will still work and new cramfs images are mountable on > old kernels if you avoid using holes (requires 2.3.39 or later) and > you don't pad the beginning of the cramfs filesystem for a boot sector > (and presumably, you have a new kernel in that case). > > Most of this code has been fairly well-tested, but a few features are > newer and less well-tested (like the directory sorting). > > If you use cramfs, please try it out and let me know if you have any > feedback or comments (please Cc me if you respond here). > > Dan > I've read the patch, and the added benefits of sorting seem usefull. I have some general questions with regard to direction for Cramfs while the subject is broached however. On the Agenda we use cramfs as our storage mechanism, with some linear addressing patches since it's on flash, and the memory copies inherent with a block device are unnecesary in our case. (And we kill the buffer of compressed data in cramfs, as it's unnecessary as well) Going forward though we're working on some other modifications of the the cramfs filesystem that allow it todo some other stuff that requires an expansion of the inode, Linus mentions in the README for cramfs that it may be usefull to expand it, but I wanted to get peoples reactions on expanding it by 32bits, and using a part of that to denote file information like type of compression, etc. I had worked previously on hacking in LZO compression into cramfs, but hadn't had time to finish when called onto other things. The other thing we're working on with ROMDISK is putting in XIP mode for binaries/libs that we're going to leave uncompressed, and aligned so they don't need to be copied to memory before executing. I'd like to integrate XIP into cramfs at some point to allow the flash releases from agenda to be released as a single unit, but doing so necesitates an expansion of the inode to denote this information about certain executables/libs. So what do you all think of expanding the inode for this purpose, and is loss of backward compatibility a massivly evil thing, so when I make this patch I need to keep information based on the magic of the cramfs filesystem to denote versioning information to allow for the difference in the inode. Thanks, Shane. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: cramfs filesystem patch
On Friday 08 December 2000 05:11, Daniel Quinlan wrote: Here's a patch for the cramfs filesystem. Lots of improvements and a new cramfsck program, see below for the full list of changes. It only modifies cramfs code (aside from adding cramfs to struct super_block) and aims to be completely backward-compatible. All old cramfs images will still work and new cramfs images are mountable on old kernels if you avoid using holes (requires 2.3.39 or later) and you don't pad the beginning of the cramfs filesystem for a boot sector (and presumably, you have a new kernel in that case). Most of this code has been fairly well-tested, but a few features are newer and less well-tested (like the directory sorting). If you use cramfs, please try it out and let me know if you have any feedback or comments (please Cc me if you respond here). Dan I've read the patch, and the added benefits of sorting seem usefull. I have some general questions with regard to direction for Cramfs while the subject is broached however. On the Agenda we use cramfs as our storage mechanism, with some linear addressing patches since it's on flash, and the memory copies inherent with a block device are unnecesary in our case. (And we kill the buffer of compressed data in cramfs, as it's unnecessary as well) Going forward though we're working on some other modifications of the the cramfs filesystem that allow it todo some other stuff that requires an expansion of the inode, Linus mentions in the README for cramfs that it may be usefull to expand it, but I wanted to get peoples reactions on expanding it by 32bits, and using a part of that to denote file information like type of compression, etc. I had worked previously on hacking in LZO compression into cramfs, but hadn't had time to finish when called onto other things. The other thing we're working on with ROMDISK is putting in XIP mode for binaries/libs that we're going to leave uncompressed, and aligned so they don't need to be copied to memory before executing. I'd like to integrate XIP into cramfs at some point to allow the flash releases from agenda to be released as a single unit, but doing so necesitates an expansion of the inode to denote this information about certain executables/libs. So what do you all think of expanding the inode for this purpose, and is loss of backward compatibility a massivly evil thing, so when I make this patch I need to keep information based on the magic of the cramfs filesystem to denote versioning information to allow for the difference in the inode. Thanks, Shane. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: cramfs filesystem patch
I'd like to see these patches as well. They may be useful on the iPAQ (and similar hardware like my Yopy here... ;-) I wish some hardware vendor out there would build an x86 box that used memory addressable flash from 0 up and RAM up higher. A simple Linux kernel boot loader could then replace the BIOS. Flash would be less expensive than 8086 lower memory windowed flash hacks. Has any work been done on e2compr in 2.4 yet? Shane Nay wrote: On Friday 08 December 2000 05:11, Daniel Quinlan wrote: Here's a patch for the cramfs filesystem. Lots of improvements and a new cramfsck program, see below for the full list of changes. It only modifies cramfs code (aside from adding cramfs to struct super_block) and aims to be completely backward-compatible. All old cramfs images will still work and new cramfs images are mountable on old kernels if you avoid using holes (requires 2.3.39 or later) and you don't pad the beginning of the cramfs filesystem for a boot sector (and presumably, you have a new kernel in that case). Most of this code has been fairly well-tested, but a few features are newer and less well-tested (like the directory sorting). If you use cramfs, please try it out and let me know if you have any feedback or comments (please Cc me if you respond here). Dan I've read the patch, and the added benefits of sorting seem usefull. I have some general questions with regard to direction for Cramfs while the subject is broached however. On the Agenda we use cramfs as our storage mechanism, with some linear addressing patches since it's on flash, and the memory copies inherent with a block device are unnecesary in our case. (And we kill the buffer of compressed data in cramfs, as it's unnecessary as well) Going forward though we're working on some other modifications of the the cramfs filesystem that allow it todo some other stuff that requires an expansion of the inode, Linus mentions in the README for cramfs that it may be usefull to expand it, but I wanted to get peoples reactions on expanding it by 32bits, and using a part of that to denote file information like type of compression, etc. I had worked previously on hacking in LZO compression into cramfs, but hadn't had time to finish when called onto other things. The other thing we're working on with ROMDISK is putting in XIP mode for binaries/libs that we're going to leave uncompressed, and aligned so they don't need to be copied to memory before executing. I'd like to integrate XIP into cramfs at some point to allow the flash releases from agenda to be released as a single unit, but doing so necesitates an expansion of the inode to denote this information about certain executables/libs. So what do you all think of expanding the inode for this purpose, and is loss of backward compatibility a massivly evil thing, so when I make this patch I need to keep information based on the magic of the cramfs filesystem to denote versioning information to allow for the difference in the inode. Thanks, Shane. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/ -- Tim Riker - http://rikers.org/ - short SIGs! g All I need to know I could have learned in Kindergarten ... if I'd just been paying attention. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
cramfs filesystem patch
Here's a patch for the cramfs filesystem. Lots of improvements and a
new cramfsck program, see below for the full list of changes.
It only modifies cramfs code (aside from adding cramfs to struct
super_block) and aims to be completely backward-compatible. All old
cramfs images will still work and new cramfs images are mountable on
old kernels if you avoid using holes (requires 2.3.39 or later) and
you don't pad the beginning of the cramfs filesystem for a boot sector
(and presumably, you have a new kernel in that case).
Most of this code has been fairly well-tested, but a few features are
newer and less well-tested (like the directory sorting).
If you use cramfs, please try it out and let me know if you have any
feedback or comments (please Cc me if you respond here).
Dan
superblock changes:
- revised fsid field (still unique, contains useful fs info:
a CRC, edition number for implementation-specific versioning,
block and file counts for statfs)
- size field is now used
- new feature flags (fsid version 2, holes, shifted root for boot loader,
directory sorting)
inode.c
- proper checking for device size (torvalds)
- speed up IO a bit - read in the block asynchronously rather than
using bread() on them one by one (torvalds)
- early directory lookup exit (since directories are now sorted)
- add reporting of blocks and blksize for statfs
- removed superfluous test for cramfs signature
cramfsck.c
- new program to fsck/extract cramfs images
mkcramfs.c
- CRC added to filesystem image
- allow holes to be created (see -z option)
- support for a cramfs boot loader (see -p option)
- support for inserting an kernel image into a cramfs image (see -i option)
- directory sorting (more consistent images, allows faster lookups)
- bug fix: allocate enough RAM for the fs image (small images)
- bug fix: work around a bug in ramfs where it would
incorrectly report the number of blocks for file (a problem when
creating cramfs images from a ramfs filesystem)
- options to set name/edition number in superblock (-n and -e, respectively)
cramfs.txt:
- added entry for /etc/magic to cramfs.txt documentation
cleanup:
- moved width definitions to cramfs.h
- define special cramfs super block
diff -ur linux-2.4.0-test10.orig/Documentation/filesystems/cramfs.txt
linux-2.4.0-test10/Documentation/filesystems/cramfs.txt
--- linux-2.4.0-test10.orig/Documentation/filesystems/cramfs.txtSat May 20
11:30:31 2000
+++ linux-2.4.0-test10/Documentation/filesystems/cramfs.txt Wed Dec 6 19:12:12
+2000
@@ -47,6 +47,21 @@
mind the filesystem becoming unreadable to future kernels.
+For /usr/share/magic
+--
+
+0 long0x28cd3d45 Linux cramfs
+>4 longx size %d
+>8 longx flags 0x%x
+>12longx future 0x%x
+>16string >\0 signature "%.16s"
+>32longx fsid.crc 0x%x
+>36longx fsid.edition %d
+>40longx fsid.blocks %d
+>44longx fsid.files %d
+>48string >\0 name "%.16s"
+
+
Hacker Notes
diff -ur linux-2.4.0-test10.orig/fs/cramfs/cramfs.h
linux-2.4.0-test10/fs/cramfs/cramfs.h
--- linux-2.4.0-test10.orig/fs/cramfs/cramfs.h Tue Jan 11 10:24:58 2000
+++ linux-2.4.0-test10/fs/cramfs/cramfs.h Wed Dec 6 19:12:12 2000
@@ -5,12 +5,23 @@
#define CRAMFS_SIGNATURE "Compressed ROMFS"
/*
+ * Width of various bitfields in struct cramfs_inode.
+ * Primarily used to generate warnings in mkcramfs.
+ */
+#define CRAMFS_MODE_WIDTH 16
+#define CRAMFS_UID_WIDTH 16
+#define CRAMFS_SIZE_WIDTH 24
+#define CRAMFS_GID_WIDTH 8
+#define CRAMFS_NAMELEN_WIDTH 6
+#define CRAMFS_OFFSET_WIDTH 26
+
+/*
* Reasonably terse representation of the inode data.
*/
struct cramfs_inode {
- u32 mode:16, uid:16;
+ u32 mode:CRAMFS_MODE_WIDTH, uid:CRAMFS_UID_WIDTH;
/* SIZE for device files is i_rdev */
- u32 size:24, gid:8;
+ u32 size:CRAMFS_SIZE_WIDTH, gid:CRAMFS_GID_WIDTH;
/* NAMELEN is the length of the file name, divided by 4 and
rounded up. (cramfs doesn't support hard links.) */
/* OFFSET: For symlinks and non-empty regular files, this
@@ -19,7 +30,14 @@
see README). For non-empty directories it is the offset
(divided by 4) of the inode of the first file in that
directory. For anything else, offset is zero. */
- u32 namelen:6, offset:26;
+ u32 namelen:CRAMFS_NAMELEN_WIDTH, offset:CRAMFS_OFFSET_WIDTH;
+};
+
+struct cramfs_info {
+ u32 crc;
+ u32 edition;
+ u32 blocks;
+ u32 files;
};
/*
@@ -27,22 +45,33 @@
*/
struct cramfs_super {
u32 magic; /* 0x28cd3d45 - random number */
- u32 size; /* Not used. mkcramfs
cramfs filesystem patch
Here's a patch for the cramfs filesystem. Lots of improvements and a
new cramfsck program, see below for the full list of changes.
It only modifies cramfs code (aside from adding cramfs to struct
super_block) and aims to be completely backward-compatible. All old
cramfs images will still work and new cramfs images are mountable on
old kernels if you avoid using holes (requires 2.3.39 or later) and
you don't pad the beginning of the cramfs filesystem for a boot sector
(and presumably, you have a new kernel in that case).
Most of this code has been fairly well-tested, but a few features are
newer and less well-tested (like the directory sorting).
If you use cramfs, please try it out and let me know if you have any
feedback or comments (please Cc me if you respond here).
Dan
superblock changes:
- revised fsid field (still unique, contains useful fs info:
a CRC, edition number for implementation-specific versioning,
block and file counts for statfs)
- size field is now used
- new feature flags (fsid version 2, holes, shifted root for boot loader,
directory sorting)
inode.c
- proper checking for device size (torvalds)
- speed up IO a bit - read in the block asynchronously rather than
using bread() on them one by one (torvalds)
- early directory lookup exit (since directories are now sorted)
- add reporting of blocks and blksize for statfs
- removed superfluous test for cramfs signature
cramfsck.c
- new program to fsck/extract cramfs images
mkcramfs.c
- CRC added to filesystem image
- allow holes to be created (see -z option)
- support for a cramfs boot loader (see -p option)
- support for inserting an kernel image into a cramfs image (see -i option)
- directory sorting (more consistent images, allows faster lookups)
- bug fix: allocate enough RAM for the fs image (small images)
- bug fix: work around a bug in ramfs where it would
incorrectly report the number of blocks for file (a problem when
creating cramfs images from a ramfs filesystem)
- options to set name/edition number in superblock (-n and -e, respectively)
cramfs.txt:
- added entry for /etc/magic to cramfs.txt documentation
cleanup:
- moved width definitions to cramfs.h
- define special cramfs super block
diff -ur linux-2.4.0-test10.orig/Documentation/filesystems/cramfs.txt
linux-2.4.0-test10/Documentation/filesystems/cramfs.txt
--- linux-2.4.0-test10.orig/Documentation/filesystems/cramfs.txtSat May 20
11:30:31 2000
+++ linux-2.4.0-test10/Documentation/filesystems/cramfs.txt Wed Dec 6 19:12:12
+2000
@@ -47,6 +47,21 @@
mind the filesystem becoming unreadable to future kernels.
+For /usr/share/magic
+--
+
+0 long0x28cd3d45 Linux cramfs
+4 longx size %d
+8 longx flags 0x%x
+12longx future 0x%x
+16string \0 signature "%.16s"
+32longx fsid.crc 0x%x
+36longx fsid.edition %d
+40longx fsid.blocks %d
+44longx fsid.files %d
+48string \0 name "%.16s"
+
+
Hacker Notes
diff -ur linux-2.4.0-test10.orig/fs/cramfs/cramfs.h
linux-2.4.0-test10/fs/cramfs/cramfs.h
--- linux-2.4.0-test10.orig/fs/cramfs/cramfs.h Tue Jan 11 10:24:58 2000
+++ linux-2.4.0-test10/fs/cramfs/cramfs.h Wed Dec 6 19:12:12 2000
@@ -5,12 +5,23 @@
#define CRAMFS_SIGNATURE "Compressed ROMFS"
/*
+ * Width of various bitfields in struct cramfs_inode.
+ * Primarily used to generate warnings in mkcramfs.
+ */
+#define CRAMFS_MODE_WIDTH 16
+#define CRAMFS_UID_WIDTH 16
+#define CRAMFS_SIZE_WIDTH 24
+#define CRAMFS_GID_WIDTH 8
+#define CRAMFS_NAMELEN_WIDTH 6
+#define CRAMFS_OFFSET_WIDTH 26
+
+/*
* Reasonably terse representation of the inode data.
*/
struct cramfs_inode {
- u32 mode:16, uid:16;
+ u32 mode:CRAMFS_MODE_WIDTH, uid:CRAMFS_UID_WIDTH;
/* SIZE for device files is i_rdev */
- u32 size:24, gid:8;
+ u32 size:CRAMFS_SIZE_WIDTH, gid:CRAMFS_GID_WIDTH;
/* NAMELEN is the length of the file name, divided by 4 and
rounded up. (cramfs doesn't support hard links.) */
/* OFFSET: For symlinks and non-empty regular files, this
@@ -19,7 +30,14 @@
see README). For non-empty directories it is the offset
(divided by 4) of the inode of the first file in that
directory. For anything else, offset is zero. */
- u32 namelen:6, offset:26;
+ u32 namelen:CRAMFS_NAMELEN_WIDTH, offset:CRAMFS_OFFSET_WIDTH;
+};
+
+struct cramfs_info {
+ u32 crc;
+ u32 edition;
+ u32 blocks;
+ u32 files;
};
/*
@@ -27,22 +45,33 @@
*/
struct cramfs_super {
u32 magic; /* 0x28cd3d45 - random number */
- u32 size; /* Not used. mkcramfs currently
-
