[PATCH 0/2] Squashfs: add LZ4 compression
Hi Now that LZ4 compression support is in 3.11-rc1, I have written the following two patches for Squashfs to use it. Phillip Lougher (2): Squashfs: add LZ4 compression support Squashfs: Add LZ4 compression configuration option Documentation/filesystems/squashfs.txt |8 +- fs/squashfs/Kconfig| 15 +++ fs/squashfs/Makefile |1 + fs/squashfs/decompressor.c |7 ++ fs/squashfs/decompressor.h |4 + fs/squashfs/lz4_wrapper.c | 163 fs/squashfs/squashfs_fs.h |1 + 7 files changed, 195 insertions(+), 4 deletions(-) create mode 100644 fs/squashfs/lz4_wrapper.c These patches are also available in the git tree here: browse: https://git.kernel.org/cgit/linux/kernel/git/pkl/squashfs-lz4.git git clone: git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-lz4.git LZ4 support has (obviously) also been added to the squashfs-tools (Mksquashfs and Unsquashfs). This is available from the Squashfs-tools git repository here: browse: https://git.kernel.org/cgit/fs/squashfs/squashfs-tools.git git clone: git://git.kernel.org/pub/scm/fs/squashfs/squashfs-tools.git When building the squashfs-tools edit the Makefile to enable LZ4 support (by default it is disabled). LZ4 compression can be specified by using the -comp option, e.g. % mksquashfs xxx img.sqsh -comp lz4 The use of LZ4 high compression can be specified using -Xhc, e.g. % mksquashfs xxx img.sqsh -comp lz4 -Xhc Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/2] Squashfs: Add LZ4 compression configuration option
Add the glue code, and also update the documentation.
Signed-off-by: Phillip Lougher phil...@squashfs.org.uk
---
Documentation/filesystems/squashfs.txt |8
fs/squashfs/Kconfig| 15 +++
fs/squashfs/Makefile |1 +
fs/squashfs/decompressor.c |7 +++
fs/squashfs/decompressor.h |4
5 files changed, 31 insertions(+), 4 deletions(-)
diff --git a/Documentation/filesystems/squashfs.txt
b/Documentation/filesystems/squashfs.txt
index 403c090..e5274f8 100644
--- a/Documentation/filesystems/squashfs.txt
+++ b/Documentation/filesystems/squashfs.txt
@@ -2,10 +2,10 @@ SQUASHFS 4.0 FILESYSTEM
===
Squashfs is a compressed read-only filesystem for Linux.
-It uses zlib/lzo/xz compression to compress files, inodes and directories.
-Inodes in the system are very small and all blocks are packed to minimise
-data overhead. Block sizes greater than 4K are supported up to a maximum
-of 1Mbytes (default block size 128K).
+It uses zlib, lz4, lzo, or xz compression to compress files, inodes and
+directories. Inodes in the system are very small and all blocks are packed to
+minimise data overhead. Block sizes greater than 4K are supported up to a
+maximum of 1Mbytes (default block size 128K).
Squashfs is intended for general read-only filesystem use, for archival
use (i.e. in cases where a .tar.gz file may be used), and in constrained
diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
index c70111e..257f934 100644
--- a/fs/squashfs/Kconfig
+++ b/fs/squashfs/Kconfig
@@ -48,6 +48,21 @@ config SQUASHFS_ZLIB
If unsure, say Y.
+config SQUASHFS_LZ4
+ bool Include support for LZ4 compressed file systems
+ depends on SQUASHFS
+ select LZ4_DECOMPRESS
+ help
+ Saying Y here includes support for reading Squashfs file systems
+ compressed with LZ4 compression. LZ4 compression is mainly
+ aimed at embedded systems with slower CPUs where the overheads
+ of zlib are too high.
+
+ LZ4 is not the standard compression used in Squashfs and so most
+ file systems will be readable without selecting this option.
+
+ If unsure, say N.
+
config SQUASHFS_LZO
bool Include support for LZO compressed file systems
depends on SQUASHFS
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index 110b047..4a80ca7 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -6,6 +6,7 @@ obj-$(CONFIG_SQUASHFS) += squashfs.o
squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o
squashfs-y += namei.o super.o symlink.o decompressor.o
squashfs-$(CONFIG_SQUASHFS_XATTR) += xattr.o xattr_id.o
+squashfs-$(CONFIG_SQUASHFS_LZ4) += lz4_wrapper.o
squashfs-$(CONFIG_SQUASHFS_LZO) += lzo_wrapper.o
squashfs-$(CONFIG_SQUASHFS_XZ) += xz_wrapper.o
squashfs-$(CONFIG_SQUASHFS_ZLIB) += zlib_wrapper.o
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
index 3f6271d..ae60211 100644
--- a/fs/squashfs/decompressor.c
+++ b/fs/squashfs/decompressor.c
@@ -40,6 +40,12 @@ static const struct squashfs_decompressor
squashfs_lzma_unsupported_comp_ops = {
NULL, NULL, NULL, LZMA_COMPRESSION, lzma, 0
};
+#ifndef CONFIG_SQUASHFS_LZ4
+static const struct squashfs_decompressor squashfs_lz4_comp_ops = {
+ NULL, NULL, NULL, LZ4_COMPRESSION, lz4, 0
+};
+#endif
+
#ifndef CONFIG_SQUASHFS_LZO
static const struct squashfs_decompressor squashfs_lzo_comp_ops = {
NULL, NULL, NULL, LZO_COMPRESSION, lzo, 0
@@ -64,6 +70,7 @@ static const struct squashfs_decompressor
squashfs_unknown_comp_ops = {
static const struct squashfs_decompressor *decompressor[] = {
squashfs_zlib_comp_ops,
+ squashfs_lz4_comp_ops,
squashfs_lzo_comp_ops,
squashfs_xz_comp_ops,
squashfs_lzma_unsupported_comp_ops,
diff --git a/fs/squashfs/decompressor.h b/fs/squashfs/decompressor.h
index 330073e..a9fbdd2 100644
--- a/fs/squashfs/decompressor.h
+++ b/fs/squashfs/decompressor.h
@@ -52,6 +52,10 @@ static inline int squashfs_decompress(struct
squashfs_sb_info *msblk,
extern const struct squashfs_decompressor squashfs_xz_comp_ops;
#endif
+#ifdef CONFIG_SQUASHFS_LZ4
+extern const struct squashfs_decompressor squashfs_lz4_comp_ops;
+#endif
+
#ifdef CONFIG_SQUASHFS_LZO
extern const struct squashfs_decompressor squashfs_lzo_comp_ops;
#endif
--
1.7.10.4
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Squashfs: add LZ4 compression
On 22 July 2013 04:05, Gu Zheng guz.f...@cn.fujitsu.com wrote: Hi Phillip, Have some tests been carried out to confirm that Squashfs really can get benefit from LZ4 compression, comparing with lzo? This seems to be a loaded question, in that it seems to be trying to reopen the why add lz4 when we already have lzo debate all over again. As LZ4 has been merged to mainline, this appears to be a question that has already been answered. As far as Squashfs is concerned, I believe it is important to give people the choice of using LZ4 to compress Squashfs filesystems now its been mainlined. As far as expected benefits are concerned, Squashfs' use in embedded systems is very similar to compressing kernels and initramfs data, in that it tends to be used to compress root filesystems. As such the benefits of using LZ4 in Squashfs should be broadly similar to using LZ4 to compress kernels and initramfs data. Ultimately it is up to people to experiment and choose whatever compression is best for their systems. People are welcome to try the patches out and report their findings. Phillip Thanks, Gu On 07/22/2013 10:21 AM, Phillip Lougher wrote: Hi Now that LZ4 compression support is in 3.11-rc1, I have written the following two patches for Squashfs to use it. Phillip Lougher (2): Squashfs: add LZ4 compression support Squashfs: Add LZ4 compression configuration option Documentation/filesystems/squashfs.txt |8 +- fs/squashfs/Kconfig| 15 +++ fs/squashfs/Makefile |1 + fs/squashfs/decompressor.c |7 ++ fs/squashfs/decompressor.h |4 + fs/squashfs/lz4_wrapper.c | 163 fs/squashfs/squashfs_fs.h |1 + 7 files changed, 195 insertions(+), 4 deletions(-) create mode 100644 fs/squashfs/lz4_wrapper.c These patches are also available in the git tree here: browse: https://git.kernel.org/cgit/linux/kernel/git/pkl/squashfs-lz4.git git clone: git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-lz4.git LZ4 support has (obviously) also been added to the squashfs-tools (Mksquashfs and Unsquashfs). This is available from the Squashfs-tools git repository here: browse: https://git.kernel.org/cgit/fs/squashfs/squashfs-tools.git git clone: git://git.kernel.org/pub/scm/fs/squashfs/squashfs-tools.git When building the squashfs-tools edit the Makefile to enable LZ4 support (by default it is disabled). LZ4 compression can be specified by using the -comp option, e.g. % mksquashfs xxx img.sqsh -comp lz4 The use of LZ4 high compression can be specified using -Xhc, e.g. % mksquashfs xxx img.sqsh -comp lz4 -Xhc Phillip -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] Squashfs: Add XZ compression configuration option
Lasse Collin wrote: On 2010-12-09 Phillip Lougher wrote: +config SQUASHFS_XZ + bool Include support for XZ compressed file systems + depends on SQUASHFS + default n + select XZ_DEC Should select XZ_DEC be replaced with depends on XZ_DEC? XZ_DEC requires CRC32, so if select XZ_DEC is used, there needs to be also select CRC32. XZ_DEC selects CRC32, kbuild handles these nested selects quite happily, so if something selects XZ_DEC it knows it has to also select CRC32. Depends on has quite different semantics to selects. If SQUASHFS_XZ was made to depend on XZ_DEC then the option simply won't appear unless the user knew to select XZ_DEC first (as it's default n). This would prove extremely confusing, and probably lead to most people thinking Squashfs didn't have XZ support, which is somewhat undesirable. XZ_DEC may optionally use other XZ_DEC_* symbols, which the user will want to choose when building for an embedded system. With depends on XZ_DEC the user will see that there's more than a single option that affects the details of the XZ support in Squashfs. With depends on XZ_DEC the user will simply not see that Squashfs has XZ support (as the option won't appear unless XZ_DEC is explicitly selected). With selects XZ_DEC users will see that Squashfs has XZ support, if enabled, they'll simply see that XZ_DEC has been automatically selected in the Library routines sub-menu. If EMBEDDED is not selected the XZ_DEC options will be automatically selected (as they're only user selectable if EMBEDDED is selected). If EMBEDDED is selected, then they'll have the choice then to decide which options they wish to de-select. I think this is preferable to needing XZ_DEC to be selected before the SQUASHFS_XZ option even appears. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/2] Squashfs: add XZ compression support
Hi, Following the recent posting of patches by Lasse Collin to add XZ (LZMA2) support to the kernel (http://thread.gmane.org/gmane.linux.kernel/1071297), I have added support for this to Squashfs. Advantages of Squashfs XZ over the Squashfs LZMA implementation: - Significantly better decompressor API supporting multi-call decoding, which requires less buffer overhead. - Greater data robustness due to XZ's CRC32 check. - BCJ filters which can produce smaller Squashfs images. The following two patches add Squashfs kernel support. A git tree with these patches including Lasse Collin's patches is available here: http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-xz.git;a=summary XZ support has (obviously) also been added to the squashfs tools (Mksquashfs Unsquashfs). These changes are available from the Squashfs CVS repository (http://sourceforge.net/projects/squashfs/develop). To build the Squashfs tools, edit the Makefile to enable XZ support (by default it is disabled). XZ compression can be specified by using the -comp option, e.g. % mksquashfs xxx img.sqsh -comp xz XZ BCJ filters (which can improve the compression of executable code on certain architectures) are supported by using the -Xbcj option, e.g. % mksquashfs xxx img.sqsh -comp xz -Xbcj x86 will compress blocks using XZ with no filter, and then XZ with the x86 filter in turn, and choose the best compression. Multiple filters can be specified which is useful in cases where the source file system has executable code from a mixture of different architectures, and again each filter will be tried for each block and the best compression used, e.g. % mksquashfs xxx img.sqsh -comp xz -Xbcj x86,arm will try both the x86 and arm BCJ filters. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/2] Squashfs: add XZ compression support
Add XZ decompressor wrapper code.
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/squashfs_fs.h |1 +
fs/squashfs/xz_wrapper.c | 153 +
2 files changed, 154 insertions(+), 0 deletions(-)
create mode 100644 fs/squashfs/xz_wrapper.c
diff --git a/fs/squashfs/squashfs_fs.h b/fs/squashfs/squashfs_fs.h
index c5137fc..39533fe 100644
--- a/fs/squashfs/squashfs_fs.h
+++ b/fs/squashfs/squashfs_fs.h
@@ -238,6 +238,7 @@ struct meta_index {
#define ZLIB_COMPRESSION 1
#define LZMA_COMPRESSION 2
#define LZO_COMPRESSION3
+#define XZ_COMPRESSION 4
struct squashfs_super_block {
__le32 s_magic;
diff --git a/fs/squashfs/xz_wrapper.c b/fs/squashfs/xz_wrapper.c
new file mode 100644
index 000..053fe35
--- /dev/null
+++ b/fs/squashfs/xz_wrapper.c
@@ -0,0 +1,153 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * xz_wrapper.c
+ */
+
+
+#include linux/mutex.h
+#include linux/buffer_head.h
+#include linux/slab.h
+#include linux/xz.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+#include decompressor.h
+
+struct squashfs_xz {
+ struct xz_dec *state;
+ struct xz_buf buf;
+};
+
+static void *squashfs_xz_init(struct squashfs_sb_info *msblk)
+{
+int block_size = max_t(int, msblk-block_size, SQUASHFS_METADATA_SIZE);
+
+struct squashfs_xz *stream = kmalloc(sizeof(*stream), GFP_KERNEL);
+if (stream == NULL)
+goto failed;
+ stream-state = xz_dec_init(XZ_PREALLOC, block_size);
+ if (stream-state == NULL)
+ goto failed;
+
+ return stream;
+
+failed:
+ ERROR(Failed to allocate xz workspace\n);
+ kfree(stream);
+ return NULL;
+}
+
+
+static void squashfs_xz_free(void *strm)
+{
+ struct squashfs_xz *stream = strm;
+
+ if (stream) {
+ xz_dec_end(stream-state);
+ kfree(stream);
+ }
+}
+
+
+static int squashfs_xz_uncompress(struct squashfs_sb_info *msblk, void
**buffer,
+ struct buffer_head **bh, int b, int offset, int length, int srclength,
+ int pages)
+{
+ enum xz_ret xz_err;
+ int avail, total = 0, k = 0, page = 0;
+ struct squashfs_xz *stream = msblk-stream;
+
+ mutex_lock(msblk-read_data_mutex);
+
+ xz_dec_reset(stream-state);
+ stream-buf.in_pos = 0;
+ stream-buf.in_size = 0;
+ stream-buf.out_pos = 0;
+ stream-buf.out_size = PAGE_CACHE_SIZE;
+ stream-buf.out = buffer[page++];
+
+ do {
+ if (stream-buf.in_pos == stream-buf.in_size k b) {
+ avail = min(length, msblk-devblksize - offset);
+ length -= avail;
+ wait_on_buffer(bh[k]);
+ if (!buffer_uptodate(bh[k]))
+ goto release_mutex;
+
+ if (avail == 0) {
+ offset = 0;
+ put_bh(bh[k++]);
+ continue;
+ }
+
+ stream-buf.in = bh[k]-b_data + offset;
+ stream-buf.in_size = avail;
+ stream-buf.in_pos = 0;
+ offset = 0;
+ }
+
+ if (stream-buf.out_pos == stream-buf.out_size
+page pages) {
+ stream-buf.out = buffer[page++];
+ stream-buf.out_pos = 0;
+ total += PAGE_CACHE_SIZE;
+ }
+
+ xz_err = xz_dec_run(stream-state, stream-buf);
+
+ if (stream-buf.in_pos == stream-buf.in_size k b)
+ put_bh(bh[k++]);
+ } while (xz_err == XZ_OK);
+
+ if (xz_err != XZ_STREAM_END) {
+ ERROR(xz_dec_run error, data probably corrupt\n);
+ goto release_mutex;
+ }
+
+ if (k b) {
+ ERROR(xz_uncompress error, input remaining\n
[PATCH 2/2] Squashfs: Add XZ compression configuration option
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/Kconfig| 16
fs/squashfs/Makefile |1 +
fs/squashfs/decompressor.c | 11 +++
fs/squashfs/squashfs.h |3 +++
4 files changed, 31 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
index e5f63da..e96d99a 100644
--- a/fs/squashfs/Kconfig
+++ b/fs/squashfs/Kconfig
@@ -53,6 +53,22 @@ config SQUASHFS_LZO
If unsure, say N.
+config SQUASHFS_XZ
+ bool Include support for XZ compressed file systems
+ depends on SQUASHFS
+ default n
+ select XZ_DEC
+ help
+ Saying Y here includes support for reading Squashfs file systems
+ compressed with XZ compresssion. XZ gives better compression than
+ the default zlib compression, at the expense of greater CPU and
+ memory overhead.
+
+ XZ is not the standard compression used in Squashfs and so most
+ file systems will be readable without selecting this option.
+
+ If unsure, say N.
+
config SQUASHFS_EMBEDDED
bool Additional option for memory-constrained systems
depends on SQUASHFS
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index 7672bac..cecf2be 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -7,3 +7,4 @@ squashfs-y += block.o cache.o dir.o export.o file.o fragment.o
id.o inode.o
squashfs-y += namei.o super.o symlink.o zlib_wrapper.o decompressor.o
squashfs-$(CONFIG_SQUASHFS_XATTR) += xattr.o xattr_id.o
squashfs-$(CONFIG_SQUASHFS_LZO) += lzo_wrapper.o
+squashfs-$(CONFIG_SQUASHFS_XZ) += xz_wrapper.o
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
index 24af9ce..ac333b8 100644
--- a/fs/squashfs/decompressor.c
+++ b/fs/squashfs/decompressor.c
@@ -46,6 +46,12 @@ static const struct squashfs_decompressor
squashfs_lzo_unsupported_comp_ops = {
};
#endif
+#ifndef CONFIG_SQUASHFS_XZ
+static const struct squashfs_decompressor squashfs_xz_unsupported_comp_ops = {
+ NULL, NULL, NULL, XZ_COMPRESSION, xz, 0
+};
+#endif
+
static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
NULL, NULL, NULL, 0, unknown, 0
};
@@ -58,6 +64,11 @@ static const struct squashfs_decompressor *decompressor[] = {
#else
squashfs_lzo_unsupported_comp_ops,
#endif
+#ifdef CONFIG_SQUASHFS_XZ
+ squashfs_xz_comp_ops,
+#else
+ squashfs_xz_unsupported_comp_ops,
+#endif
squashfs_unknown_comp_ops
};
diff --git a/fs/squashfs/squashfs.h b/fs/squashfs/squashfs.h
index 5d45569..1096e2e 100644
--- a/fs/squashfs/squashfs.h
+++ b/fs/squashfs/squashfs.h
@@ -107,3 +107,6 @@ extern const struct squashfs_decompressor
squashfs_zlib_comp_ops;
/* lzo_wrapper.c */
extern const struct squashfs_decompressor squashfs_lzo_comp_ops;
+
+/* xz_wrapper.c */
+extern const struct squashfs_decompressor squashfs_xz_comp_ops;
--
1.6.3.3
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC 2/3] Decompressors: Add boot-time XZ support
On 24/11/10 20:54, Lasse Collin wrote:
@@ -176,10 +179,20 @@ config KERNEL_LZMA
bool LZMA
depends on HAVE_KERNEL_LZMA
help
+ This is the predecessor of XZ.
+
You seem to have moved the help text from LZMA into the entry for XZ,
leaving LZMA merely with the observation it's the predecessor of XZ. I
think LZMA should keep it's help text describing what it is.
+config KERNEL_XZ
+ bool XZ
+ depends on HAVE_KERNEL_XZ
+ help
The most recent compression algorithm.
This sounds odd, The most recent compression algorithm to what? The
most recent compression algorithm in the world, the most recent open
source compression algorithm, or the most recently added compression
algorithm to the Linux kernel?
These statements can become quickly out of date, especially if they're
vague as to what they mean - if someone added another compression
algorithm to the Linux kernel in the future, should they change this
statement?
BTW I appreciate that you've merely kept the original poorly worded
description from the LZMA help text.
diff -uprN linux-2.6.37-rc3.orig/lib/decompress_unxz.c
linux-2.6.37-rc3/lib/decompress_unxz.c
--- linux-2.6.37-rc3.orig/lib/decompress_unxz.c 1970-01-01 02:00:00.0
+0200
+++ linux-2.6.37-rc3/lib/decompress_unxz.c 2010-11-24 18:18:37.0
+0200
@@ -0,0 +1,390 @@
+/*
+ * XZ decoder as a single file for uncompressing the kernel and initramfs
Does Single file XZ decoder for ... sound better?
+#ifndef memeq
+static bool memeq(const void *a, const void *b, size_t size)
+{
+ const uint8_t *x = a;
+ const uint8_t *y = b;
+ size_t i;
+
+ for (i = 0; i size; ++i)
The kernel uses either i size or isize ...
lots of these ...
+ if (x[i] != y[i])
+ return false;
+
+ return true;
+}
+#endif
+
+#ifndef memzero
+static void memzero(void *buf, size_t size)
+{
+ uint8_t *b = buf;
+ uint8_t *e = b + size;
New line here
+ while (b != e)
+ *b++ = '\0';
+}
+#endif
+
+/*
+ * This function implements the API defined inlinux/decompress/generic.h.
+ *
Not completely, see below. Your wrapper behaves correctly (bar one
respect) for the restricted use cases of initramfs/initrd, but for
other inputs it will behave differently to the other decompressors, and
differently to that described in generic.h, and it is broken for some
inputs.
Is this a problem? Depends on your viewpoint. One viewpoint is all
the decompressors/wrappers should behave the same (as realistically
possible) given the same inputs, so code can switch between compressors
and get the same behaviour. The other viewpoint is to just say that
the decompressors give the same behaviour for the restricted
inittramfs/initrd use cases and state all other usage is unpredictable.
+ if (in != NULL out != NULL)
+ s = xz_dec_init(XZ_SINGLE, 0);
+ else
+ s = xz_dec_init(XZ_DYNALLOC, (uint32_t)-1);
+
+ if (s == NULL)
+ goto error_alloc_state;
+
+ b.in = in;
+ b.in_pos = 0;
+ b.in_size = in_size;
+ b.out_pos = 0;
+
+ if (in_used != NULL)
+ *in_used = 0;
+
+ if (fill == NULL flush == NULL) {
Space before
+ b.out = out;
+ b.out_size = (size_t)-1;
+ ret = xz_dec_run(s,b);
+ } else {
+ b.out_size = XZ_IOBUF_SIZE;
+ b.out = malloc(XZ_IOBUF_SIZE);
You're assuming here that flush != NULL. The API as described in
generic.h allows for the situation where fill != NULL flush == NULL,
in which case out is assumed to be != NULL and large enough to hold the
entire output data without flushing.
From generic.h: If flush = NULL, outbuf must be large enough to buffer
all the expected output
+ if (b.out == NULL)
+ goto error_alloc_out;
+
+ if (fill != NULL) {
+ in = malloc(XZ_IOBUF_SIZE);
From generic.h: inbuf can be NULL, in which case the decompressor will
allocate the input buffer. If inbuf != NULL it must be at least
XXX_IOBUF_SIZE bytes. fill will be called (repeatedly...) to read data
If in != NULL, you'll discard the passed in buffer.
+ if (in == NULL)
+ goto error_alloc_in;
+
+ b.in = in;
+ }
+
+ do {
+ if (b.in_pos == b.in_size fill != NULL) {
Space before
If fill != NULL you're relying on the caller to have passed
in_size == 0, so first time around the loop the fill function is
called to fill your empty malloced buffer. If in_size is passed in
!= 0, you won't call fill and therefore you will pass an empty buffer
to the decompressor.
+ if (in_used != NULL)
+ *in_used += b.in_pos;
+
+
+
Re: [PATCH RFC 2/3] Decompressors: Add boot-time XZ support
On 25/11/10 06:32, Phillip Lougher wrote:
+
+ for (i = 0; i size; ++i)
The kernel uses either i size or isize ...
Disregard these, looks like my email client (Mozilla Thunderbird)
is eating spaces... Very odd
+ if (fill == NULL flush == NULL) {
and this.
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[ANN] Squashfs tools 4.1 released
Hi, I'm pleased to announce the release of Squashfs tools 4.1. This release adds support for extended attributes (XATTRs), and LZMA and LZO compression. There's also new pseudo file features and the usual bug fixes. The release can be downloaded from http://sourceforge.net/projects/squashfs. Compatibility Mksquashfs 4.1 generates 4.0 filesystems. These filesystems are fully compatible/interchangable with filesystems generated by Mksquashfs 4.0 and are mountable on 2.6.29 and later kernels. Extended attributes (xattrs) Squashfs file systems now have extended attribute support. The extended attribute implementation has the following features: 1. Layout can store up to 248 bytes of compressed xattr data. 2. Number of xattrs per inode unlimited. 3. Total size of xattr data per inode 248 bytes of compressed data. 4. Up to 4 Gbytes of data per xattr value. 5. Inline and out-of-line xattr values supported for higher performance in xattr scanning (listxattr getxattr), and to allow xattr value de-duplication. 6. Both whole inode xattr duplicate detection and individual xattr value duplicate detection supported. These can obviously nest, file C's xattrs can be a complete duplicate of file B, and file B's xattrs can be a partial duplicate of file A. 7. Xattr name prefix types stored, allowing the redundant user., trusted. etc. characters to be eliminated and more concisely stored. 8. Support for files, directories, symbolic links, device nodes, fifos and sockets. Extended attribute support is in 2.6.35 and later kernels. File systems with extended attributes can be mounted on 2.6.29 and later kernels, the extended attributes will be ignored with a warning. LZMA and LZO compression Squashfs now supports LZMA and LZO compression. LZO support is in 2.6.36 and newer kernels. LZMA is not yet in mainline. New pseudo file support --- Mksquashfs supports pseudo files, these allow fake files, directories, character and block devices to be specified and added to the Squashfs filesystem being built, rather than requiring them to be present in the source directories. This, for example, allows device nodes to be added to the filesystem without requiring root access. Mksquashfs 4.1 adds support for dynamic pseudo files and a modify operation. Dynamic pseudo files allow files to be dynamically created when Mksquashfs is run, their contents being the result of running a command or piece of shell script. The modifiy operation allows the mode/uid/gid of an existing file in the source filesystem to be modified. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Squashfs extended attribute file system support
Hi, As promised way back in 2008, I have finished xattr support for Squashfs. Xattr layout highlights 1. Layout can store up to 2^48 bytes of compressed xattr data. 2. Number of xattrs per inode unlimited. 3. Total size of xattr data per inode 2^48 bytes of compressed data. 4. Up to 4 Gbytes of data per xattr value. 5. Inline and out-of-line xattr values supported for higher performance in xattr scanning (listxattr getxattr), and to allow xattr value de-duplication. 6. Both whole inode xattr duplicate detection and individual xattr value duplicate detection supported. These can obviously nest, file C's xattrs can be a complete duplicate of file B, and file B's xattrs can be a partial duplicate of file A. 7. Xattr name prefix types stored, allowing the redundant user., trusted. etc. characters to be eliminated and more concisely stored. 8. Support for files, directories, symbolic links, device nodes, fifos and sockets. Xattr support for mksquashfs is now in squashfs CVS. Support for xattrs in mksquashfs append mode, and in Unsquashfs will be finished ASAP. The latest version of mksquashfs in CVS can be obtained here http://sourceforge.net/projects/squashfs/develop Kernel xattr support is in git, here git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-xattr.git master gitweb http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-xattr.git;a=summary Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RFC: direct MTD support for SquashFS
Ferenc Wagner wrote: Now with the patch series, sorry. Ferenc Wagner wf...@niif.hu writes: I've got one more patch, which I forgot to export, to pull out the common logic from the backend init functions back into squashfs_read_data(). With the bdev backend, that entails reading the first block twice in a row most of the time. This again could be worked around by extending the backend interface, but I'm not sure if it's worth it. Here it is. I also corrected the name of SQUASHFS_METADATA_SIZE, so it may as well compile now. Thanks for your patches. First things first. Patch sending etiquette :-) 1. Please don't send separate git commits in one big email, it makes them hard to cross-reference (I lost count of the times I had to repeatedly scroll though the email to check an earlier commit). 2. Please don't send a patch series consisting of your development process. Reviewing takes effort, having reviewed one commit to find it reworked in a second commit, and then reworked again, as the code evolves from a - b - c- d is irritating, it makes the final solution harder to evaluate (because you have to keep all the changes in your head), and wastes my time. 3. Incremental patches are valid if they break up a patch series into easily digestible changes which can be separately understood, but not if they're just reworking your code as development progresses... OK, now for the specific comments. Frameworks are supposed to make the code cleaner, more understandable, and generally better. Unfortunately, I see no evidence of that in your patch. Sorry to be blunt, but there it is. The backend has seven functions! + void*(*init)(struct squashfs_sb_info *, u64, size_t); + void(*free)(struct squashfs_sb_info *); + ssize_t (*read)(struct squashfs_sb_info *, void **, size_t); + int (*probe)(struct file_system_type *, int, const char *, + void*, struct vfsmount *); + void(*kill)(struct squashfs_sb_info *); + loff_t (*size)(const struct super_block *); + const char *(*devname)(const struct super_block *, char *); We move from a single read() function to requiring seven functions to do the same work, just to add mtd support... Reading the superblock changes into a 6 function deep nightmare squashfs_find_backend - probe - get_sb_dev - fill_bdev_super - squashfs_fill_super - add_backend I find the current 3 function deep situation forced by VFS already a mess, squashfs_get_sb - get_sb_bdev - squashfs_fill_super Reading a block now takes 3 function calls, init, read, and free. Plus in your last commit you make the huge mistake of preferring code elegance over performance, and resort to calling init, read, and free *twice*, the first just to read *two* bytes (yes, 2 bytes). Why do you think the code was coded that way in the first place? A fit of absent mindedness perhaps? No, for performance reasons. Secondly you make the classic mistake of concentrating on what you want to do (add MTD), and not giving a damn about anything else. You've totally broken all the read optimisations in zlib decompression for block devices. Buffer_heads are deliberately passed directly to the zlib decompressor to allow it to optimise performance (pass the buffer_head directly to zlib etc.). Buffer_heads are exposed to the decompressors without crappy go-between wrappers deliberately. Unfortunately there are lots of other instances where you've eliminated carefully optimised code. That's another of my major issues, the patch seems hugely gratuitous, it touches a huge amount of files/code it shouldn't do, much of this slicing up optimisied heavily tested and fragile code into other files/functions hither and thither. Unfortunately much of this code took a long time to get right, and suffered numerous unanticipated edge conditions/fsfuzzer triggered bugs. I only touch this code with caution and nothing like to the extent you've subjected it to. In short this doesn't pass my quality threshold or the make the minimum changes necessary threshold. I wouldn't consider asking Linus to merge this. BTW as a comparison, I have added MTD support to Squashfs twice in the past, *with* bad block support. The latest has a diff patch of ~500 lines, with far less complexity and code changes necessary. BTW2 as evidenced by your FIXME comments against my code in your patch, you really have to get over the notion that if you don't understand it, the code must be wrong. Cheers Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RFC: direct MTD support for SquashFS
On Fri, Mar 19, 2010 at 1:05 AM, Ferenc Wagner wf...@niif.hu wrote:
Ferenc Wagner wf...@niif.hu writes:
Phillip Lougher phillip.loug...@gmail.com writes:
On Thu, Mar 18, 2010 at 4:38 PM, Ferenc Wagner wf...@niif.hu wrote:
+static int fill_bdev_super(struct super_block *sb, void *data, int silent)
+{
+ struct squashfs_sb_info *msblk;
+ struct squashfs_bdev *bdev;
+ int err = squashfs_fill_super2(sb, data, silent, squashfs_bdev_ops);
+ if (err)
+ return err;
+
+ bdev = kzalloc(sizeof(*bdev), GFP_KERNEL);
+ if (!bdev)
+ return -ENOMEM;
+
+ bdev-devblksize = sb_min_blocksize(sb, BLOCK_SIZE);
+ bdev-devblksize_log2 = ffz(~bdev-devblksize);
+
+ msblk = sb-s_fs_info;
+ msblk-backend_data = bdev;
+ return 0;
+}
This function looks rather 'back-to-front' to me. I'm assuming that
squashfs_fill_super2() will be the current fill superblock function?
Yes, with the extra parameter added.
This function wants to read data off the filesystem through the
backend, and yet the backend (bdev, mblk-backend_data) hasn't been
initialised when it's called...
It can't be, because msblk = sb-s_fs_info is allocated by
squashfs_fill_super(). Now it will be passed the ops, so after
allocating msblk it can also fill out the ops. After that it can read,
and squashfs_read_data() will call the init, read and free operations of
the backend.
And here we indeed have a rather fundamental problem. This isn't
specific to the discussed plugin system at all. Even in the current
code, to set msblk-block_size squashfs_fill_super() calls
squashfs_read_table() to read the superblock, which in turn calls
squashfs_read_data(), which uses msblk-block_size to allocate enough
buffer heads, but msblk-block_size just can't be set at this point.
msblk-bytes_used is preset with a dummy value to make the read
possible, but msblk-block_size is not. Fortunately, one buffer head is
allocated each time nevertheless. I wonder what a correct solution
would look lke..
Block_size is known to be zero (the structure has been zeroed out at
alloc), and so it is known that the one block alloced in this case
will be correct.
Congratulations you've managed to really piss me off in your third or so email.
Cheers
Phillip
--
Regards,
Feri.
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RFC: direct MTD support for SquashFS
On Thu, Mar 18, 2010 at 4:38 PM, Ferenc Wagner wf...@niif.hu wrote:
I could only compare apples to oranges before porting the patch to the
LZMA variant. So I refrain from that for a couple of days yet. But
meanwhile I started adding a pluggable backend framework to SquashFS,
and would much appreciate some comments about the applicability of this
idea. The patch is (intended to be) a no-op, applies on top of current
git (a3d3203e4bb40f253b1541e310dc0f9305be7c84).
This looks promising, making the backend pluggable (like the new
compressor framework) is far better and cleaner than scattering the
code full of #ifdef's. Far better than the previous patch :-)
A couple of specific comments...
+/* A backend is initialized for each SquashFS block read operation,
+ * making further sequential reads possible from the block.
+ */
+static void *bdev_init(struct squashfs_sb_info *msblk, u64 index,
size_t length)
+{
+ struct squashfs_bdev *bdev = msblk-backend_data;
+ struct buffer_head *bh;
+
+ bh = kcalloc((msblk-block_size bdev-devblksize_log2) + 1,
+ sizeof(*bh), GFP_KERNEL);
You should alloc against the larger of msblk-block_size and
METADATA_SIZE (8 Kbytes). Block_size could be 4 Kbytes only.
+static int fill_bdev_super(struct super_block *sb, void *data, int silent)
+{
+ struct squashfs_sb_info *msblk;
+ struct squashfs_bdev *bdev;
+ int err = squashfs_fill_super2(sb, data, silent, squashfs_bdev_ops);
+ if (err)
+ return err;
+
+ bdev = kzalloc(sizeof(*bdev), GFP_KERNEL);
+ if (!bdev)
+ return -ENOMEM;
+
+ bdev-devblksize = sb_min_blocksize(sb, BLOCK_SIZE);
+ bdev-devblksize_log2 = ffz(~bdev-devblksize);
+
+ msblk = sb-s_fs_info;
+ msblk-backend_data = bdev;
+ return 0;
+}
This function looks rather 'back-to-front' to me. I'm assuming that
squashfs_fill_super2() will be the current fill superblock function?
This function wants to read data off the filesystem through the
backend, and yet the backend (bdev, mblk-backend_data) hasn't been
initialised when it's called...
Phillip
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V2 6/8] lzma: Make lzma available to non initramfs/initrd code
Andrew Morton wrote: So what to do? I guess I could go ahead with the mainline merge, and Stephen drops whatever that tree was from linux-next until it has been fixed up? Yes, I'm happy with that. The following patch is the necessary fix that needs to go into linux-next when you've gone ahead with the mainline merge of the lib-add-support-for-lzo-compressed-kernels.patch. Once I know the mainline merge has gone ahead, I can add this to my linux-next tree. Phillip From 1cf6d32e1427398368ff189aece68aa533092e98 Mon Sep 17 00:00:00 2001 From: Phillip Lougher phil...@lougher.demon.co.uk Date: Wed, 6 Jan 2010 23:50:12 + Subject: [PATCH] lzo: Fix-up add support for lzo compressed kernels patch The add support for lzo compressed kernels patch relies on INIT and error definitions which have been moved to separate xxx_mm.h files for each decompressor. This patch adds a unlzo_mm.h file which supplies these definitions. Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk --- include/linux/decompress/unlzo_mm.h | 13 + lib/decompress_unlzo.c |1 + 2 files changed, 14 insertions(+), 0 deletions(-) create mode 100644 include/linux/decompress/unlzo_mm.h diff --git a/include/linux/decompress/unlzo_mm.h b/include/linux/decompress/unlzo_mm.h new file mode 100644 index 000..27fe0ab --- /dev/null +++ b/include/linux/decompress/unlzo_mm.h @@ -0,0 +1,13 @@ +#ifndef UNLZO_MM_H +#define UNLZO_MM_H + +#ifdef STATIC +/* Code active when included from pre-boot environment: */ +#define INIT +#else +/* Compile for initramfs/initrd code only */ +#define INIT __init +static void(*error)(char *m); +#endif + +#endif diff --git a/lib/decompress_unlzo.c b/lib/decompress_unlzo.c index db521f4..edd82c3 100644 --- a/lib/decompress_unlzo.c +++ b/lib/decompress_unlzo.c @@ -39,6 +39,7 @@ #include linux/types.h #include linux/lzo.h +#include linux/decompress/unlzo_mm.h #include linux/decompress/mm.h #include linux/compiler.h -- 1.6.3.3 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V2 6/8] lzma: Make lzma available to non initramfs/initrd code
Fix up patch resend using a nice and dumb email client. If I'm lucky, email readers may even get the threading right :-) Phillip From 1cf6d32e1427398368ff189aece68aa533092e98 Mon Sep 17 00:00:00 2001 From: Phillip Lougher phil...@lougher.demon.co.uk Date: Wed, 6 Jan 2010 23:50:12 + Subject: [PATCH] lzo: Fix-up add support for lzo compressed kernels patch The add support for lzo compressed kernels patch relies on INIT and error definitions which have been moved to separate xxx_mm.h files for each decompressor. This patch adds a unlzo_mm.h file which supplies these definitions. Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk --- include/linux/decompress/unlzo_mm.h | 13 + lib/decompress_unlzo.c |1 + 2 files changed, 14 insertions(+), 0 deletions(-) create mode 100644 include/linux/decompress/unlzo_mm.h diff --git a/include/linux/decompress/unlzo_mm.h b/include/linux/decompress/unlzo_mm.h new file mode 100644 index 000..27fe0ab --- /dev/null +++ b/include/linux/decompress/unlzo_mm.h @@ -0,0 +1,13 @@ +#ifndef UNLZO_MM_H +#define UNLZO_MM_H + +#ifdef STATIC +/* Code active when included from pre-boot environment: */ +#define INIT +#else +/* Compile for initramfs/initrd code only */ +#define INIT __init +static void(*error)(char *m); +#endif + +#endif diff --git a/lib/decompress_unlzo.c b/lib/decompress_unlzo.c index db521f4..edd82c3 100644 --- a/lib/decompress_unlzo.c +++ b/lib/decompress_unlzo.c @@ -39,6 +39,7 @@ #include linux/types.h #include linux/lzo.h +#include linux/decompress/unlzo_mm.h #include linux/decompress/mm.h #include linux/compiler.h -- 1.6.3.3 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/9] Squashfs: Add support for LZMA compressed filesystems
Bruno Wolff III wrote: On Mon, Dec 07, 2009 at 08:37:48 +, Phillip Lougher phil...@lougher.demon.co.uk wrote: Hi, The following patches add LZMA decompression support to Squashfs, using the in-kernel LZMA decompression library. Do these still have a chance to make 2.6.33? It's very unlikely. Andrew wanted it to spend a cycle in linux-next. I'll then try for 2.6.34 merge. I'm happy with that. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/3] bzip2/lzma/gzip/initramfs: NULL pointer bugs and missing error checks
Hi, The following patches fix up a number of NULL pointer bugs amd missing error checks. Thanks Phillip bzip2/lzma/gzip: pre-boot malloc doesn't return NULL on failure bzip2: Add missing checks for malloc returning NULL initramfs: add missing decompressor error check --- include/linux/decompress/mm.h |4 ++-- init/initramfs.c | 10 ++ lib/decompress_bunzip2.c | 10 +- 3 files changed, 17 insertions(+), 7 deletions(-) -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/3] initramfs: add missing decompressor error check
The decompressors return error by calling a supplied error function, and/or
by returning an error return value. The initramfs code, however, fails to
check the exit code returned by the decompressor, and only checks the error
status set by calling the error function.
This patch adds a return code check and calls the error function.
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
init/initramfs.c | 10 ++
1 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/init/initramfs.c b/init/initramfs.c
index 4c00edc..b37d34b 100644
--- a/init/initramfs.c
+++ b/init/initramfs.c
@@ -413,7 +413,7 @@ static unsigned my_inptr; /* index of next byte to be
processed in inbuf */
static char * __init unpack_to_rootfs(char *buf, unsigned len)
{
- int written;
+ int written, res;
decompress_fn decompress;
const char *compress_name;
static __initdata char msg_buf[64];
@@ -445,10 +445,12 @@ static char * __init unpack_to_rootfs(char *buf, unsigned
len)
}
this_header = 0;
decompress = decompress_method(buf, len, compress_name);
- if (decompress)
- decompress(buf, len, NULL, flush_buffer, NULL,
+ if (decompress) {
+ res = decompress(buf, len, NULL, flush_buffer, NULL,
my_inptr, error);
- else if (compress_name) {
+ if (res)
+ error(decompressor failed);
+ } else if (compress_name) {
if (!message) {
snprintf(msg_buf, sizeof msg_buf,
compression method %s not configured,
--
1.6.3.3
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
Geert Uytterhoeven wrote: Yes I did consider using the cryptoapi, but this doesn't have support for lzma in mainline. IIRC, Felix Fietkau added support for that for OpenWRT... Yes, but it isn't in mainline, and OpenWRT don't appear to have tried to submit it. IMHO the major problem with their patch is it uses a second private copy of lzma, rather than being a wrapper around the pre-existing lzma implementation. I don't think it's going to be easy to get a second lzma implementation accepted into mainline, when it took so long to get one accepted. I have nothing against the cryptoapi, but it doesn't seem likely to be getting lzma support anytime soon. As I previously said my aim is to use the pre-existing lzma implementation. Unless it is stupendously bad (which it isn't), that seems to be the quickest, easiest and best way to get lzma support into Squashfs. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
Geert Uytterhoeven wrote: Hi Phillip, On Thu, Dec 10, 2009 at 01:38, Phillip Lougher phil...@lougher.demon.co.uk wrote: Artem Bityutskiy wrote: Did you consider using cryptoapi? UBIFS uses zlib/lzo in cryptoapi - it is a very clean way. Exactly my question, as that's why the Crypto API was extended with support for partial (de)compression in the first place ;-) Your cryptoapi patches are on my must do something about them list, I've not forgotten them :-) The lack of lzma in the crypotapi made moving solely to the cryptoapi difficult, when I wanted to add lzma decompression support. The decompressor framework is my _solution_ to this problem. This allows cryptoapi support to be added as additional decompression_wrapper, with multiple compression types using the crptoapi wrapper as appropriate. This has the advantage it allows Squashfs to use the cryptoapi (with its advantages), without needlessly restricting Squashfs to the compression algorithms supported by the cryptoapi. Let's get this mainlined, and then we can work on getting your cryptoapi patches integrated into that. That'll make me happy, and hopefully you'll be happy too? Thanks Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V2 0/9] Squashfs: Add support for LZMA compressed filesystems
Hi, This a respin of my LZMA decompression support patches, taking into account Andrew Morton's comments regarding poorly named globals: Changes: 1. zlib_uncompress() renamed to squashfs_zlib_uncompress() (patch 1) 2. zlib_init() and zlib_free() renamed to squashfs_zlib_init() and squashfs_zlib_free() (patch 2) 3. Merged patch 7 (make decompressor init function pass superblock info) into patch 3. This was a development commit which should not have been a separate patch for review Original patch series info: The following patches add LZMA decompression support to Squashfs, using the in-kernel LZMA decompression library. The patches also add a decompression framework to Squashfs. This allows LZMA decompression to be added cleanly, and it allows additional decompressors to be easily added in the future. To enable the in-kernel LZMA decompression code to be used by Squashfs, there are two patches to the lzma code itself: one to make lzma available to non-init code, and one to make lzma reentrant. These are obviously not restricted to Squashfs, but are needed by any non-init code that may wish to use lzma compression. These patches are available in my squashfs-devel git tree in a slightly different format (these patches have been refactored for posting, they'll be put into a git tree ASAP). http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-devel.git;a=summary I would like to thank the CE Linux Forum (CELF) for supporting this work. Thanks Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V2 3/8] Squashfs: add a decompressor framework
This adds a decompressor framework which allows multiple compression
algorithms to be cleanly supported.
Also update zlib wrapper and other code to use the new framework.
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/Makefile |2 +-
fs/squashfs/block.c |3 +-
fs/squashfs/decompressor.c | 58 ++
fs/squashfs/decompressor.h | 55 +++
fs/squashfs/squashfs.h | 14 +-
fs/squashfs/squashfs_fs_sb.h | 41 +++--
fs/squashfs/super.c | 45 ++-
fs/squashfs/zlib_wrapper.c | 17 ++--
8 files changed, 184 insertions(+), 51 deletions(-)
create mode 100644 fs/squashfs/decompressor.c
create mode 100644 fs/squashfs/decompressor.h
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index a397e6f..df8a19e 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -4,4 +4,4 @@
obj-$(CONFIG_SQUASHFS) += squashfs.o
squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o
-squashfs-y += namei.o super.o symlink.o zlib_wrapper.o
+squashfs-y += namei.o super.o symlink.o zlib_wrapper.o decompressor.o
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
index 3f836e1..1cb0d81 100644
--- a/fs/squashfs/block.c
+++ b/fs/squashfs/block.c
@@ -36,6 +36,7 @@
#include squashfs_fs_sb.h
#include squashfs_fs_i.h
#include squashfs.h
+#include decompressor.h
/*
* Read the metadata block length, this is stored in the first two
@@ -151,7 +152,7 @@ int squashfs_read_data(struct super_block *sb, void
**buffer, u64 index,
}
if (compressed) {
- length = squashfs_zlib_uncompress(msblk, buffer, bh, b, offset,
+ length = squashfs_decompress(msblk, buffer, bh, b, offset,
length, srclength, pages);
if (length 0)
goto read_failure;
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
new file mode 100644
index 000..0072ccd
--- /dev/null
+++ b/fs/squashfs/decompressor.c
@@ -0,0 +1,58 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * decompressor.c
+ */
+
+#include linux/types.h
+#include linux/mutex.h
+#include linux/buffer_head.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include decompressor.h
+#include squashfs.h
+
+/*
+ * This file (and decompressor.h) implements a decompressor framework for
+ * Squashfs, allowing multiple decompressors to be easily supported
+ */
+
+static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
+ NULL, NULL, NULL, 0, unknown, 0
+};
+
+static const struct squashfs_decompressor *decompressor[] = {
+ squashfs_zlib_comp_ops,
+ squashfs_unknown_comp_ops
+};
+
+
+const struct squashfs_decompressor *squashfs_lookup_decompressor(int id)
+{
+ int i;
+
+ for (i = 0; decompressor[i]-id; i++)
+ if (id == decompressor[i]-id)
+ break;
+
+ return decompressor[i];
+}
diff --git a/fs/squashfs/decompressor.h b/fs/squashfs/decompressor.h
new file mode 100644
index 000..7425f80
--- /dev/null
+++ b/fs/squashfs/decompressor.h
@@ -0,0 +1,55 @@
+#ifndef DECOMPRESSOR_H
+#define DECOMPRESSOR_H
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write
[PATCH V2 7/8] Squashfs: select DECOMPRESS_LZMA_NEEDED when including support for lzma
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk --- fs/squashfs/Kconfig |1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig index 0294aa2..7ec5d7e 100644 --- a/fs/squashfs/Kconfig +++ b/fs/squashfs/Kconfig @@ -30,6 +30,7 @@ config SQUASHFS_LZMA bool Include support for LZMA compressed file systems depends on SQUASHFS select DECOMPRESS_LZMA + select DECOMPRESS_LZMA_NEEDED config SQUASHFS_EMBEDDED -- 1.6.3.3 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V2 1/8] Squashfs: move zlib decompression wrapper code into a separate file
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/Makefile |2 +-
fs/squashfs/block.c| 74 ++
fs/squashfs/squashfs.h |4 ++
fs/squashfs/zlib_wrapper.c | 109
4 files changed, 118 insertions(+), 71 deletions(-)
create mode 100644 fs/squashfs/zlib_wrapper.c
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index 70e3244..a397e6f 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -4,4 +4,4 @@
obj-$(CONFIG_SQUASHFS) += squashfs.o
squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o
-squashfs-y += namei.o super.o symlink.o
+squashfs-y += namei.o super.o symlink.o zlib_wrapper.o
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
index 2a79603..b8addfd 100644
--- a/fs/squashfs/block.c
+++ b/fs/squashfs/block.c
@@ -29,7 +29,6 @@
#include linux/fs.h
#include linux/vfs.h
#include linux/slab.h
-#include linux/mutex.h
#include linux/string.h
#include linux/buffer_head.h
#include linux/zlib.h
@@ -153,72 +152,10 @@ int squashfs_read_data(struct super_block *sb, void
**buffer, u64 index,
}
if (compressed) {
- int zlib_err = 0, zlib_init = 0;
-
- /*
-* Uncompress block.
-*/
-
- mutex_lock(msblk-read_data_mutex);
-
- msblk-stream.avail_out = 0;
- msblk-stream.avail_in = 0;
-
- bytes = length;
- do {
- if (msblk-stream.avail_in == 0 k b) {
- avail = min(bytes, msblk-devblksize - offset);
- bytes -= avail;
- wait_on_buffer(bh[k]);
- if (!buffer_uptodate(bh[k]))
- goto release_mutex;
-
- if (avail == 0) {
- offset = 0;
- put_bh(bh[k++]);
- continue;
- }
-
- msblk-stream.next_in = bh[k]-b_data + offset;
- msblk-stream.avail_in = avail;
- offset = 0;
- }
-
- if (msblk-stream.avail_out == 0 page pages) {
- msblk-stream.next_out = buffer[page++];
- msblk-stream.avail_out = PAGE_CACHE_SIZE;
- }
-
- if (!zlib_init) {
- zlib_err = zlib_inflateInit(msblk-stream);
- if (zlib_err != Z_OK) {
- ERROR(zlib_inflateInit returned
-unexpected result 0x%x,
-srclength %d\n, zlib_err,
- srclength);
- goto release_mutex;
- }
- zlib_init = 1;
- }
-
- zlib_err = zlib_inflate(msblk-stream, Z_SYNC_FLUSH);
-
- if (msblk-stream.avail_in == 0 k b)
- put_bh(bh[k++]);
- } while (zlib_err == Z_OK);
-
- if (zlib_err != Z_STREAM_END) {
- ERROR(zlib_inflate error, data probably corrupt\n);
- goto release_mutex;
- }
-
- zlib_err = zlib_inflateEnd(msblk-stream);
- if (zlib_err != Z_OK) {
- ERROR(zlib_inflate error, data probably corrupt\n);
- goto release_mutex;
- }
- length = msblk-stream.total_out;
- mutex_unlock(msblk-read_data_mutex);
+ length = squashfs_zlib_uncompress(msblk, buffer, bh, b, offset,
+length, srclength, pages);
+ if (length 0)
+ goto read_failure;
} else {
/*
* Block is uncompressed.
@@ -255,9 +192,6 @@ int squashfs_read_data(struct super_block *sb, void
**buffer, u64 index,
kfree(bh);
return length;
-release_mutex:
- mutex_unlock(msblk-read_data_mutex);
-
block_release:
for (; k b; k++)
put_bh(bh[k]);
diff --git a/fs/squashfs/squashfs.h b/fs/squashfs/squashfs.h
index 0e9feb6..ba87db6 100644
--- a/fs/squashfs/squashfs.h
+++ b/fs/squashfs/squashfs.h
@@ -70,6 +70,10 @@ extern struct inode *squashfs_iget(struct super_block *,
long long,
unsigned int);
extern int squashfs_read_inode(struct inode *, long long);
+/* zlib_wrapper.c */
+extern int squashfs_zlib_uncompress(struct squashfs_sb_info *, void
[PATCH V2 5/8] Squashfs: add support for LZMA compressed filesystems
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/Kconfig|5 ++
fs/squashfs/Makefile |1 +
fs/squashfs/decompressor.c |4 +
fs/squashfs/lzma_wrapper.c | 151
fs/squashfs/squashfs.h |3 +
5 files changed, 164 insertions(+), 0 deletions(-)
create mode 100644 fs/squashfs/lzma_wrapper.c
diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
index 25a00d1..0294aa2 100644
--- a/fs/squashfs/Kconfig
+++ b/fs/squashfs/Kconfig
@@ -26,6 +26,11 @@ config SQUASHFS
If unsure, say N.
+config SQUASHFS_LZMA
+ bool Include support for LZMA compressed file systems
+ depends on SQUASHFS
+ select DECOMPRESS_LZMA
+
config SQUASHFS_EMBEDDED
bool Additional option for memory-constrained systems
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index df8a19e..45aaefd 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -5,3 +5,4 @@
obj-$(CONFIG_SQUASHFS) += squashfs.o
squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o
squashfs-y += namei.o super.o symlink.o zlib_wrapper.o decompressor.o
+squashfs-$(CONFIG_SQUASHFS_LZMA) += lzma_wrapper.o
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
index 157478d..0b6ad9b 100644
--- a/fs/squashfs/decompressor.c
+++ b/fs/squashfs/decompressor.c
@@ -50,7 +50,11 @@ static const struct squashfs_decompressor
squashfs_unknown_comp_ops = {
static const struct squashfs_decompressor *decompressor[] = {
squashfs_zlib_comp_ops,
+#ifdef CONFIG_SQUASHFS_LZMA
+ squashfs_lzma_comp_ops,
+#else
squashfs_lzma_unsupported_comp_ops,
+#endif
squashfs_lzo_unsupported_comp_ops,
squashfs_unknown_comp_ops
};
diff --git a/fs/squashfs/lzma_wrapper.c b/fs/squashfs/lzma_wrapper.c
new file mode 100644
index 000..cef06d6
--- /dev/null
+++ b/fs/squashfs/lzma_wrapper.c
@@ -0,0 +1,151 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * lzma_wrapper.c
+ */
+
+#include asm/unaligned.h
+#include linux/buffer_head.h
+#include linux/mutex.h
+#include linux/vmalloc.h
+#include linux/decompress/unlzma.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+#include decompressor.h
+
+struct squashfs_lzma {
+ void*input;
+ void*output;
+};
+
+/* decompress_unlzma.c is currently non re-entrant... */
+DEFINE_MUTEX(lzma_mutex);
+
+/* decompress_unlzma.c doesn't provide any context in its callbacks... */
+static int lzma_error;
+
+static void error(char *m)
+{
+ ERROR(unlzma error: %s\n, m);
+ lzma_error = 1;
+}
+
+
+static void *lzma_init(struct squashfs_sb_info *msblk)
+{
+ struct squashfs_lzma *stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+ if (stream == NULL)
+ goto failed;
+ stream-input = vmalloc(msblk-block_size);
+ if (stream-input == NULL)
+ goto failed;
+ stream-output = vmalloc(msblk-block_size);
+ if (stream-output == NULL)
+ goto failed2;
+
+ return stream;
+
+failed2:
+ vfree(stream-input);
+failed:
+ ERROR(failed to allocate lzma workspace\n);
+ kfree(stream);
+ return NULL;
+}
+
+
+static void lzma_free(void *strm)
+{
+ struct squashfs_lzma *stream = strm;
+
+ if (stream) {
+ vfree(stream-input);
+ vfree(stream-output);
+ }
+ kfree(stream);
+}
+
+
+static int lzma_uncompress(struct squashfs_sb_info *msblk, void **buffer,
+ struct buffer_head **bh, int b, int offset, int length, int srclength,
+ int pages)
+{
+ struct squashfs_lzma *stream = msblk-stream;
+ void *buff = stream-input;
+ int avail, i, bytes = length, res;
+
+ mutex_lock(lzma_mutex);
+
+ for (i = 0; i b; i++) {
+ wait_on_buffer(bh[i]);
+ if (!buffer_uptodate(bh[i]))
+ goto block_release;
+
+ avail = min(bytes, msblk-devblksize - offset);
+ memcpy(buff, bh[i]-b_data + offset, avail);
+ buff
Re: [PATCH 0/9] Squashfs: Add support for LZMA compressed filesystems
Peter Korsgaard wrote: root == root r...@lougher.demon.co.uk writes: root Hi, root The following patches add LZMA decompression support to Squashfs, using the root in-kernel LZMA decompression library. root The patches also add a decompression framework to Squashfs. root This allows LZMA decompression to be added cleanly, and it allows root additional decompressors to be easily added in the future. root To enable the in-kernel LZMA decompression code to be used by Squashfs, root there are two patches to the lzma code itself: one to make lzma available root to non-init code, and one to make lzma reentrant. These are obviously root not restricted to Squashfs, but are needed by any non-init code that root may wish to use lzma compression. Nice, do you also have an mksquashfs with lzma support? Yes, in Squashfs CVS (http://sourceforge.net/projects/squashfs/develop). You'll need to edit the squashfs-tools Makefile to enable LZMA support. The comments in the Makefile should, hopefully, explain how to build LZMA support into Mksquashfs/Unsquashfs. Once built-in, LZMA support can be specified using the -comp lzma option, i.e. mksquashfs dir dir.img -comp lzma Unsquashfs doesn't need any extra options, it automatically detects which compression has been used. You can tell which compression algorithms Mksquashfs/Unsquashfs support by just typing the command on its own (i.e. mksquashfs, or unsquashfs). The (de-)compressors available are displayed at the end of the output. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 07/9] lzma: Make lzma available to non initramfs/initrd code
Add a config option DECOMPRESS_LZMA_NEEDED which allows subsystems to specify they need the unlzma code. Normally decompress_unlzma.c is compiled with __init and unlzma is not exported to modules. Move INIT definition into separate header files for bzip2/lzma/inflate so it can be defined differently for each decompressor. Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk --- include/linux/decompress/bunzip2_mm.h | 12 include/linux/decompress/inflate_mm.h | 12 include/linux/decompress/mm.h |3 --- include/linux/decompress/unlzma_mm.h | 20 lib/Kconfig |3 +++ lib/decompress_bunzip2.c |1 + lib/decompress_inflate.c |1 + lib/decompress_unlzma.c |6 +- 8 files changed, 54 insertions(+), 4 deletions(-) create mode 100644 include/linux/decompress/bunzip2_mm.h create mode 100644 include/linux/decompress/inflate_mm.h create mode 100644 include/linux/decompress/unlzma_mm.h diff --git a/include/linux/decompress/bunzip2_mm.h b/include/linux/decompress/bunzip2_mm.h new file mode 100644 index 000..cac6fef --- /dev/null +++ b/include/linux/decompress/bunzip2_mm.h @@ -0,0 +1,12 @@ +#ifndef BUNZIP2_MM_H +#define BUNZIP2_MM_H + +#ifdef STATIC +/* Code active when included from pre-boot environment: */ +#define INIT +#else +/* Compile for initramfs/initrd code only */ +#define INIT __init +#endif + +#endif diff --git a/include/linux/decompress/inflate_mm.h b/include/linux/decompress/inflate_mm.h new file mode 100644 index 000..ca4a2ae --- /dev/null +++ b/include/linux/decompress/inflate_mm.h @@ -0,0 +1,12 @@ +#ifndef INFLATE_MM_H +#define INFLATE_MM_H + +#ifdef STATIC +/* Code active when included from pre-boot environment: */ +#define INIT +#else +/* Compile for initramfs/initrd code only */ +#define INIT __init +#endif + +#endif diff --git a/include/linux/decompress/mm.h b/include/linux/decompress/mm.h index 12ff8c3..80f5ba4 100644 --- a/include/linux/decompress/mm.h +++ b/include/linux/decompress/mm.h @@ -53,8 +53,6 @@ static void free(void *where) #define set_error_fn(x) -#define INIT - #else /* STATIC */ /* Code active when compiled standalone for use when loading ramdisk: */ @@ -77,7 +75,6 @@ static void free(void *where) static void(*error)(char *m); #define set_error_fn(x) error = x; -#define INIT __init #define STATIC #include linux/init.h diff --git a/include/linux/decompress/unlzma_mm.h b/include/linux/decompress/unlzma_mm.h new file mode 100644 index 000..859287e --- /dev/null +++ b/include/linux/decompress/unlzma_mm.h @@ -0,0 +1,20 @@ +#ifndef UNLZMA_MM_H +#define UNLZMA_MM_H + +#ifdef STATIC + +/* Code active when included from pre-boot environment: */ +#define INIT + +#elif defined(CONFIG_DECOMPRESS_LZMA_NEEDED) + +/* Make it available to non initramfs/initrd code */ +#define INIT +#include linux/module.h +#else + +/* Compile for initramfs/initrd code only */ +#define INIT __init +#endif + +#endif diff --git a/lib/Kconfig b/lib/Kconfig index bb1326d..25e7f28 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -117,6 +117,9 @@ config DECOMPRESS_BZIP2 config DECOMPRESS_LZMA tristate +config DECOMPRESS_LZMA_NEEDED +boolean + # # Generic allocator support is selected if needed # diff --git a/lib/decompress_bunzip2.c b/lib/decompress_bunzip2.c index 600f473..6eb6433 100644 --- a/lib/decompress_bunzip2.c +++ b/lib/decompress_bunzip2.c @@ -52,6 +52,7 @@ #include linux/slab.h #endif /* STATIC */ +#include linux/decompress/bunzip2_mm.h #include linux/decompress/mm.h #ifndef INT_MAX diff --git a/lib/decompress_inflate.c b/lib/decompress_inflate.c index fc686c7..cb6bcab 100644 --- a/lib/decompress_inflate.c +++ b/lib/decompress_inflate.c @@ -23,6 +23,7 @@ #endif /* STATIC */ +#include linux/decompress/inflate_mm.h #include linux/decompress/mm.h #define GZIP_IOBUF_SIZE (16*1024) diff --git a/lib/decompress_unlzma.c b/lib/decompress_unlzma.c index ca82fde..a614b26 100644 --- a/lib/decompress_unlzma.c +++ b/lib/decompress_unlzma.c @@ -36,6 +36,7 @@ #include linux/slab.h #endif /* STATIC */ +#include linux/decompress/unlzma_mm.h #include linux/decompress/mm.h #defineMIN(a, b) (((a) (b)) ? (a) : (b)) @@ -531,7 +532,7 @@ static inline void INIT process_bit1(struct writer *wr, struct rc *rc, -STATIC inline int INIT unlzma(unsigned char *buf, int in_len, +STATIC int INIT unlzma(unsigned char *buf, int in_len, int(*fill)(void*, unsigned int), int(*flush)(void*, unsigned int), unsigned char *output, @@ -652,6 +653,9 @@ exit_1: exit_0: return ret; } +#if defined(CONFIG_DECOMPRESS_LZMA_NEEDED) !defined(PREBOOT) +EXPORT_SYMBOL(unlzma); +#endif #ifdef PREBOOT STATIC int INIT decompress(unsigned char *buf, int in_len, -- 1.6.3.3 -- To unsubscribe from this list: send
[PATCH 8/9] Squashfs: select DECOMPRESS_LZMA_NEEDED when including support for lzma
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk --- fs/squashfs/Kconfig |1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig index 0294aa2..7ec5d7e 100644 --- a/fs/squashfs/Kconfig +++ b/fs/squashfs/Kconfig @@ -30,6 +30,7 @@ config SQUASHFS_LZMA bool Include support for LZMA compressed file systems depends on SQUASHFS select DECOMPRESS_LZMA + select DECOMPRESS_LZMA_NEEDED config SQUASHFS_EMBEDDED -- 1.6.3.3 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 3/9] Squashfs: add a decompressor framework
This adds a decompressor framework which allows multiple compression
algorithms to be cleanly supported.
Also update zlib wrapper and other code to use the new framework.
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/Makefile |2 +-
fs/squashfs/block.c |6 ++--
fs/squashfs/decompressor.c | 58 ++
fs/squashfs/decompressor.h | 55 +++
fs/squashfs/squashfs.h | 14 +-
fs/squashfs/squashfs_fs_sb.h | 41 +++--
fs/squashfs/super.c | 45 ++-
fs/squashfs/zlib_wrapper.c | 17 ++--
8 files changed, 185 insertions(+), 53 deletions(-)
create mode 100644 fs/squashfs/decompressor.c
create mode 100644 fs/squashfs/decompressor.h
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index a397e6f..df8a19e 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -4,4 +4,4 @@
obj-$(CONFIG_SQUASHFS) += squashfs.o
squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o
-squashfs-y += namei.o super.o symlink.o zlib_wrapper.o
+squashfs-y += namei.o super.o symlink.o zlib_wrapper.o decompressor.o
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
index baf7624..6f9914d 100644
--- a/fs/squashfs/block.c
+++ b/fs/squashfs/block.c
@@ -36,7 +36,7 @@
#include squashfs_fs_sb.h
#include squashfs_fs_i.h
#include squashfs.h
-
+#include decompressor.h
/*
* Read the metadata block length, this is stored in the first two
* bytes of the metadata block.
@@ -151,8 +151,8 @@ int squashfs_read_data(struct super_block *sb, void
**buffer, u64 index,
}
if (compressed) {
- length = zlib_uncompress(msblk, buffer, bh, b, offset, length,
- srclength, pages);
+ length = squashfs_decompress(msblk, buffer, bh, b, offset,
+ length, srclength, pages);
if (length 0)
goto read_failure;
} else {
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
new file mode 100644
index 000..0072ccd
--- /dev/null
+++ b/fs/squashfs/decompressor.c
@@ -0,0 +1,58 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * decompressor.c
+ */
+
+#include linux/types.h
+#include linux/mutex.h
+#include linux/buffer_head.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include decompressor.h
+#include squashfs.h
+
+/*
+ * This file (and decompressor.h) implements a decompressor framework for
+ * Squashfs, allowing multiple decompressors to be easily supported
+ */
+
+static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
+ NULL, NULL, NULL, 0, unknown, 0
+};
+
+static const struct squashfs_decompressor *decompressor[] = {
+ squashfs_zlib_comp_ops,
+ squashfs_unknown_comp_ops
+};
+
+
+const struct squashfs_decompressor *squashfs_lookup_decompressor(int id)
+{
+ int i;
+
+ for (i = 0; decompressor[i]-id; i++)
+ if (id == decompressor[i]-id)
+ break;
+
+ return decompressor[i];
+}
diff --git a/fs/squashfs/decompressor.h b/fs/squashfs/decompressor.h
new file mode 100644
index 000..778760c
--- /dev/null
+++ b/fs/squashfs/decompressor.h
@@ -0,0 +1,55 @@
+#ifndef DECOMPRESSOR_H
+#define DECOMPRESSOR_H
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received
[PATCH 9/9] lzma: make lzma reentrant
The error function pointer used by lzma is global (file scope) which
prevents it being used concurrently. This patch removes the global
error pointer use, and instead passes it to all functions that need it.
The error function pointer is still used by bzip2 and inflate.
This patch moves the definition into the separate bzip2/inflate
header files. This prevents gcc from complaining about an
unused definition compiling lzma.
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
include/linux/decompress/bunzip2_mm.h |1 +
include/linux/decompress/inflate_mm.h |1 +
include/linux/decompress/mm.h |1 -
lib/decompress_unlzma.c | 82 +
4 files changed, 44 insertions(+), 41 deletions(-)
diff --git a/include/linux/decompress/bunzip2_mm.h
b/include/linux/decompress/bunzip2_mm.h
index cac6fef..863efd0 100644
--- a/include/linux/decompress/bunzip2_mm.h
+++ b/include/linux/decompress/bunzip2_mm.h
@@ -7,6 +7,7 @@
#else
/* Compile for initramfs/initrd code only */
#define INIT __init
+static void(*error)(char *m);
#endif
#endif
diff --git a/include/linux/decompress/inflate_mm.h
b/include/linux/decompress/inflate_mm.h
index ca4a2ae..87a742b 100644
--- a/include/linux/decompress/inflate_mm.h
+++ b/include/linux/decompress/inflate_mm.h
@@ -7,6 +7,7 @@
#else
/* Compile for initramfs/initrd code only */
#define INIT __init
+static void(*error)(char *m);
#endif
#endif
diff --git a/include/linux/decompress/mm.h b/include/linux/decompress/mm.h
index 80f5ba4..32651e4 100644
--- a/include/linux/decompress/mm.h
+++ b/include/linux/decompress/mm.h
@@ -72,7 +72,6 @@ static void free(void *where)
#define large_malloc(a) vmalloc(a)
#define large_free(a) vfree(a)
-static void(*error)(char *m);
#define set_error_fn(x) error = x;
#define STATIC
diff --git a/lib/decompress_unlzma.c b/lib/decompress_unlzma.c
index a614b26..3e85763 100644
--- a/lib/decompress_unlzma.c
+++ b/lib/decompress_unlzma.c
@@ -89,7 +89,7 @@ static int nofill(void *buffer, unsigned int len)
}
/* Called twice: once at startup and once in rc_normalize() */
-static void INIT rc_read(struct rc *rc)
+static void INIT rc_read(struct rc *rc, void(*error)(char *x))
{
rc-buffer_size = rc-fill((char *)rc-buffer, LZMA_IOBUF_SIZE);
if (rc-buffer_size = 0)
@@ -116,13 +116,13 @@ static inline void INIT rc_init(struct rc *rc,
rc-range = 0x;
}
-static inline void INIT rc_init_code(struct rc *rc)
+static inline void INIT rc_init_code(struct rc *rc, void(*error)(char *x))
{
int i;
for (i = 0; i 5; i++) {
if (rc-ptr = rc-buffer_end)
- rc_read(rc);
+ rc_read(rc, error);
rc-code = (rc-code 8) | *rc-ptr++;
}
}
@@ -135,32 +135,33 @@ static inline void INIT rc_free(struct rc *rc)
}
/* Called twice, but one callsite is in inline'd rc_is_bit_0_helper() */
-static void INIT rc_do_normalize(struct rc *rc)
+static void INIT rc_do_normalize(struct rc *rc, void(*error)(char *x))
{
if (rc-ptr = rc-buffer_end)
- rc_read(rc);
+ rc_read(rc, error);
rc-range = 8;
rc-code = (rc-code 8) | *rc-ptr++;
}
-static inline void INIT rc_normalize(struct rc *rc)
+static inline void INIT rc_normalize(struct rc *rc, void(*error)(char *x))
{
if (rc-range (1 RC_TOP_BITS))
- rc_do_normalize(rc);
+ rc_do_normalize(rc, error);
}
/* Called 9 times */
/* Why rc_is_bit_0_helper exists?
*Because we want to always expose (rc-code rc-bound) to optimizer
*/
-static inline uint32_t INIT rc_is_bit_0_helper(struct rc *rc, uint16_t *p)
+static inline uint32_t INIT rc_is_bit_0_helper(struct rc *rc, uint16_t *p,
+ void (*error)(char *x))
{
- rc_normalize(rc);
+ rc_normalize(rc, error);
rc-bound = *p * (rc-range RC_MODEL_TOTAL_BITS);
return rc-bound;
}
-static inline int INIT rc_is_bit_0(struct rc *rc, uint16_t *p)
+static inline int INIT rc_is_bit_0(struct rc *rc, uint16_t *p,
void(*error)(char *x))
{
- uint32_t t = rc_is_bit_0_helper(rc, p);
+ uint32_t t = rc_is_bit_0_helper(rc, p, error);
return rc-code t;
}
@@ -178,9 +179,9 @@ static inline void rc_update_bit_1(struct rc *rc, uint16_t
*p)
}
/* Called 4 times in unlzma loop */
-static int INIT rc_get_bit(struct rc *rc, uint16_t *p, int *symbol)
+static int INIT rc_get_bit(struct rc *rc, uint16_t *p, int *symbol,
void(*error)(char *x))
{
- if (rc_is_bit_0(rc, p)) {
+ if (rc_is_bit_0(rc, p, error)) {
rc_update_bit_0(rc, p);
*symbol *= 2;
return 0;
@@ -192,9 +193,9 @@ static int INIT rc_get_bit(struct rc *rc, uint16_t *p, int
*symbol)
}
/* Called once */
-static inline int INIT rc_direct_bit(struct rc *rc)
+static inline int INIT rc_direct_bit(struct rc *rc
[PATCH 4/9] Squashfs: add decompressor entries for lzma and lzo
Add knowledge of lzma/lzo compression formats to the decompressor
framework. For now these are added as unsupported. Without
these entries lzma/lzo compressed filesystems will be flagged as
having unknown compression which is undesirable.
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/decompressor.c | 10 ++
fs/squashfs/squashfs_fs.h |4 +++-
2 files changed, 13 insertions(+), 1 deletions(-)
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
index 0072ccd..157478d 100644
--- a/fs/squashfs/decompressor.c
+++ b/fs/squashfs/decompressor.c
@@ -36,12 +36,22 @@
* Squashfs, allowing multiple decompressors to be easily supported
*/
+static const struct squashfs_decompressor squashfs_lzma_unsupported_comp_ops =
{
+ NULL, NULL, NULL, LZMA_COMPRESSION, lzma, 0
+};
+
+static const struct squashfs_decompressor squashfs_lzo_unsupported_comp_ops = {
+ NULL, NULL, NULL, LZO_COMPRESSION, lzo, 0
+};
+
static const struct squashfs_decompressor squashfs_unknown_comp_ops = {
NULL, NULL, NULL, 0, unknown, 0
};
static const struct squashfs_decompressor *decompressor[] = {
squashfs_zlib_comp_ops,
+ squashfs_lzma_unsupported_comp_ops,
+ squashfs_lzo_unsupported_comp_ops,
squashfs_unknown_comp_ops
};
diff --git a/fs/squashfs/squashfs_fs.h b/fs/squashfs/squashfs_fs.h
index 283daaf..36e1604 100644
--- a/fs/squashfs/squashfs_fs.h
+++ b/fs/squashfs/squashfs_fs.h
@@ -211,7 +211,9 @@ struct meta_index {
/*
* definitions for structures on disk
*/
-#define ZLIB_COMPRESSION1
+#define ZLIB_COMPRESSION 1
+#define LZMA_COMPRESSION 2
+#define LZO_COMPRESSION3
struct squashfs_super_block {
__le32 s_magic;
--
1.6.3.3
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/9] Squashfs: move zlib decompression wrapper code into a separate file
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/Makefile |2 +-
fs/squashfs/block.c| 74 ++
fs/squashfs/squashfs.h |4 ++
fs/squashfs/zlib_wrapper.c | 109
4 files changed, 118 insertions(+), 71 deletions(-)
create mode 100644 fs/squashfs/zlib_wrapper.c
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index 70e3244..a397e6f 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -4,4 +4,4 @@
obj-$(CONFIG_SQUASHFS) += squashfs.o
squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o
-squashfs-y += namei.o super.o symlink.o
+squashfs-y += namei.o super.o symlink.o zlib_wrapper.o
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
index 2a79603..5cd3934 100644
--- a/fs/squashfs/block.c
+++ b/fs/squashfs/block.c
@@ -29,7 +29,6 @@
#include linux/fs.h
#include linux/vfs.h
#include linux/slab.h
-#include linux/mutex.h
#include linux/string.h
#include linux/buffer_head.h
#include linux/zlib.h
@@ -153,72 +152,10 @@ int squashfs_read_data(struct super_block *sb, void
**buffer, u64 index,
}
if (compressed) {
- int zlib_err = 0, zlib_init = 0;
-
- /*
-* Uncompress block.
-*/
-
- mutex_lock(msblk-read_data_mutex);
-
- msblk-stream.avail_out = 0;
- msblk-stream.avail_in = 0;
-
- bytes = length;
- do {
- if (msblk-stream.avail_in == 0 k b) {
- avail = min(bytes, msblk-devblksize - offset);
- bytes -= avail;
- wait_on_buffer(bh[k]);
- if (!buffer_uptodate(bh[k]))
- goto release_mutex;
-
- if (avail == 0) {
- offset = 0;
- put_bh(bh[k++]);
- continue;
- }
-
- msblk-stream.next_in = bh[k]-b_data + offset;
- msblk-stream.avail_in = avail;
- offset = 0;
- }
-
- if (msblk-stream.avail_out == 0 page pages) {
- msblk-stream.next_out = buffer[page++];
- msblk-stream.avail_out = PAGE_CACHE_SIZE;
- }
-
- if (!zlib_init) {
- zlib_err = zlib_inflateInit(msblk-stream);
- if (zlib_err != Z_OK) {
- ERROR(zlib_inflateInit returned
-unexpected result 0x%x,
-srclength %d\n, zlib_err,
- srclength);
- goto release_mutex;
- }
- zlib_init = 1;
- }
-
- zlib_err = zlib_inflate(msblk-stream, Z_SYNC_FLUSH);
-
- if (msblk-stream.avail_in == 0 k b)
- put_bh(bh[k++]);
- } while (zlib_err == Z_OK);
-
- if (zlib_err != Z_STREAM_END) {
- ERROR(zlib_inflate error, data probably corrupt\n);
- goto release_mutex;
- }
-
- zlib_err = zlib_inflateEnd(msblk-stream);
- if (zlib_err != Z_OK) {
- ERROR(zlib_inflate error, data probably corrupt\n);
- goto release_mutex;
- }
- length = msblk-stream.total_out;
- mutex_unlock(msblk-read_data_mutex);
+ length = zlib_uncompress(msblk, buffer, bh, b, offset, length,
+ srclength, pages);
+ if (length 0)
+ goto read_failure;
} else {
/*
* Block is uncompressed.
@@ -255,9 +192,6 @@ int squashfs_read_data(struct super_block *sb, void
**buffer, u64 index,
kfree(bh);
return length;
-release_mutex:
- mutex_unlock(msblk-read_data_mutex);
-
block_release:
for (; k b; k++)
put_bh(bh[k]);
diff --git a/fs/squashfs/squashfs.h b/fs/squashfs/squashfs.h
index 0e9feb6..988bdce 100644
--- a/fs/squashfs/squashfs.h
+++ b/fs/squashfs/squashfs.h
@@ -70,6 +70,10 @@ extern struct inode *squashfs_iget(struct super_block *,
long long,
unsigned int);
extern int squashfs_read_inode(struct inode *, long long);
+/* zlib_wrapper.c */
+extern int zlib_uncompress(struct squashfs_sb_info *, void
[PATCH 6/9] Squashfs: add support for LZMA compressed filesystems
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/Kconfig|5 ++
fs/squashfs/Makefile |1 +
fs/squashfs/decompressor.c |4 +
fs/squashfs/lzma_wrapper.c | 151
fs/squashfs/squashfs.h |3 +
5 files changed, 164 insertions(+), 0 deletions(-)
create mode 100644 fs/squashfs/lzma_wrapper.c
diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
index 25a00d1..0294aa2 100644
--- a/fs/squashfs/Kconfig
+++ b/fs/squashfs/Kconfig
@@ -26,6 +26,11 @@ config SQUASHFS
If unsure, say N.
+config SQUASHFS_LZMA
+ bool Include support for LZMA compressed file systems
+ depends on SQUASHFS
+ select DECOMPRESS_LZMA
+
config SQUASHFS_EMBEDDED
bool Additional option for memory-constrained systems
diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile
index df8a19e..45aaefd 100644
--- a/fs/squashfs/Makefile
+++ b/fs/squashfs/Makefile
@@ -5,3 +5,4 @@
obj-$(CONFIG_SQUASHFS) += squashfs.o
squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o
squashfs-y += namei.o super.o symlink.o zlib_wrapper.o decompressor.o
+squashfs-$(CONFIG_SQUASHFS_LZMA) += lzma_wrapper.o
diff --git a/fs/squashfs/decompressor.c b/fs/squashfs/decompressor.c
index 157478d..0b6ad9b 100644
--- a/fs/squashfs/decompressor.c
+++ b/fs/squashfs/decompressor.c
@@ -50,7 +50,11 @@ static const struct squashfs_decompressor
squashfs_unknown_comp_ops = {
static const struct squashfs_decompressor *decompressor[] = {
squashfs_zlib_comp_ops,
+#ifdef CONFIG_SQUASHFS_LZMA
+ squashfs_lzma_comp_ops,
+#else
squashfs_lzma_unsupported_comp_ops,
+#endif
squashfs_lzo_unsupported_comp_ops,
squashfs_unknown_comp_ops
};
diff --git a/fs/squashfs/lzma_wrapper.c b/fs/squashfs/lzma_wrapper.c
new file mode 100644
index 000..cef06d6
--- /dev/null
+++ b/fs/squashfs/lzma_wrapper.c
@@ -0,0 +1,151 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * lzma_wrapper.c
+ */
+
+#include asm/unaligned.h
+#include linux/buffer_head.h
+#include linux/mutex.h
+#include linux/vmalloc.h
+#include linux/decompress/unlzma.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+#include decompressor.h
+
+struct squashfs_lzma {
+ void*input;
+ void*output;
+};
+
+/* decompress_unlzma.c is currently non re-entrant... */
+DEFINE_MUTEX(lzma_mutex);
+
+/* decompress_unlzma.c doesn't provide any context in its callbacks... */
+static int lzma_error;
+
+static void error(char *m)
+{
+ ERROR(unlzma error: %s\n, m);
+ lzma_error = 1;
+}
+
+
+static void *lzma_init(struct squashfs_sb_info *msblk)
+{
+ struct squashfs_lzma *stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+ if (stream == NULL)
+ goto failed;
+ stream-input = vmalloc(msblk-block_size);
+ if (stream-input == NULL)
+ goto failed;
+ stream-output = vmalloc(msblk-block_size);
+ if (stream-output == NULL)
+ goto failed2;
+
+ return stream;
+
+failed2:
+ vfree(stream-input);
+failed:
+ ERROR(failed to allocate lzma workspace\n);
+ kfree(stream);
+ return NULL;
+}
+
+
+static void lzma_free(void *strm)
+{
+ struct squashfs_lzma *stream = strm;
+
+ if (stream) {
+ vfree(stream-input);
+ vfree(stream-output);
+ }
+ kfree(stream);
+}
+
+
+static int lzma_uncompress(struct squashfs_sb_info *msblk, void **buffer,
+ struct buffer_head **bh, int b, int offset, int length, int srclength,
+ int pages)
+{
+ struct squashfs_lzma *stream = msblk-stream;
+ void *buff = stream-input;
+ int avail, i, bytes = length, res;
+
+ mutex_lock(lzma_mutex);
+
+ for (i = 0; i b; i++) {
+ wait_on_buffer(bh[i]);
+ if (!buffer_uptodate(bh[i]))
+ goto block_release;
+
+ avail = min(bytes, msblk-devblksize - offset);
+ memcpy(buff, bh[i]-b_data + offset, avail);
+ buff
Re: [PATCH 1/6] lib/decompress_*: only include linux/slab.h if STATIC is not defined
Andrew Morton wrote: On Mon, 3 Aug 2009 16:58:16 +0200 Albin Tonnerre albin.tonne...@free-electrons.com wrote: These includes were added by 079effb6933f34b9b1b67b08bd4fd7fb672d16ef to fix the build when using kmemtrace. However this is not necessary when used to create a compressed kernel, and actually creates issues (brings a lot of things unavailable in the decompression environment), so don't include it if STATIC is defined. The description actually creates issues (brings a lot of things unavailable in the decompression environment) is inadequate. Please describe te problem this patch fixes more completely so that others (ie: me) can decide whether this patch is needed in 2.6.32, 2.6.31. 2.6.30, ... This patch conflicts heavily with http://userweb.kernel.org/~akpm/mmotm/broken-out/bzip2-lzma-remove-nasty-uncompressed-size-hack-in-pre-boot-environment.patch What should we do about that? What do you normally do in this situation? I'm happy to send a revised bzip2-lzma-remove-nasty-uncompressed-size-hack-in-pre-boot-environment.patch that would apply cleanly on-top of Alvin's patch, but, this will obviously create dependencies on his patch being applied. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/6] lib/decompress_*: only include linux/slab.h if STATIC is not defined
H. Peter Anvin wrote: On 08/04/2009 05:47 PM, Phillip Lougher wrote: Andrew Morton wrote: On Mon, 3 Aug 2009 16:58:16 +0200 Albin Tonnerre albin.tonne...@free-electrons.com wrote: These includes were added by 079effb6933f34b9b1b67b08bd4fd7fb672d16ef to fix the build when using kmemtrace. However this is not necessary when used to create a compressed kernel, and actually creates issues (brings a lot of things unavailable in the decompression environment), so don't include it if STATIC is defined. The description actually creates issues (brings a lot of things unavailable in the decompression environment) is inadequate. Please describe te problem this patch fixes more completely so that others (ie: me) can decide whether this patch is needed in 2.6.32, 2.6.31. 2.6.30, ... This patch conflicts heavily with http://userweb.kernel.org/~akpm/mmotm/broken-out/bzip2-lzma-remove-nasty-uncompressed-size-hack-in-pre-boot-environment.patch What should we do about that? What do you normally do in this situation? I'm happy to send a revised bzip2-lzma-remove-nasty-uncompressed-size-hack-in-pre-boot-environment.patch that would apply cleanly on-top of Alvin's patch, but, this will obviously create dependencies on his patch being applied. The general principle is that if A alone creates a more functional environment than B alone, then B should be applied on top of A, and vice versa. This is especially so if A is a stable candidate. It *sounds* like your patch is B here, but I am not sure from the description. Gosh, who wants to get into the my patch is better than yours argument. I certainly don't... My patch series cleans up the code and fixes a number of rough edges (which I expect to hit when I try to make Squashfs use the new decompression code). Albin's looks to be adding a new set of LZO functionality. Regarding the conflicting patches in question, my patch removes a hack, Albin's moves #include slab.h into code covered by #ifndef STATIC, so it doesn't pull in loads of unnecessary definitions when the file is being built in the pre-boot environment. I personally can't decide which is A or B. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/6] lib/decompress_*: only include linux/slab.h if STATIC is not defined
Andrew Morton wrote: On Wed, 05 Aug 2009 01:47:57 +0100 Phillip Lougher phil...@lougher.demon.co.uk wrote: What do you normally do in this situation? I normally fix the rejects ;) But I'd like to confirm that the two patches don't fix the same thing via different means. Lacking a full description of Albin's issues, that's hard to determine. They do appear to be unrelated. No they're not fixing the same thing. Albin's patch is moving #include slab.h inside the #ifndef STATIC ... #endif code segment. This ensures that slab.h isn't included when the file is being built in the stripped down pre-boot environment. I imagine Albin's issues is that slab.h pulls in a lot of definitions unnecessary in the pre-boot environment and which rely on things which are missing in the stripped down pre-boot environment. My changes to the #ifndef STATIC logic defines PREBOOT if STATIC is defined. My patch uses the PREBOOT definition later to define the decompress wrapper function, which is only needed in the preboot environment. i.e. #ifdef STATIC #define PREBOOT #else #include linux/decompress/unlzma.h #endif ... Lots of code ... #ifdef PREBOOT static int INIT decompress. #endif Obvious question, why doesn't my patch use STATIC here rather than PREBOOT? The header file linux/decompress/unlzma.h defines STATIC, .i.e the #ifndef STATIC case defines STATIC via an include file, which makes decisions on STATIC later in the file impossible. I'm happy to send a revised bzip2-lzma-remove-nasty-uncompressed-size-hack-in-pre-boot-environment.patch that would apply cleanly on-top of Alvin's patch, but, this will obviously create dependencies on his patch being applied. Reworked lib-decompress_-only-include-linux-slabh-if-static-is-not-defined.patch: The patch looks OK. Thanks Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Squashfs 4.0 tools (for 2.6.29) released
Hi, I'm pleased to announce the first release of the squashfs 4.0 tools. These are needed by any Squashfs users using linux 2.6.29. They are available from http://sourceforge.net/projects/squashfs The changes from the change-log are: 1. Unsquashfs improvements: 1.1 Support for 4.0 filesystems added. 1.2 Swapping macros rewritten. 1.3 Unsquashfs code restructured and split into separate files. 2. Mksquashfs improvements: 2.1 Swapping macros rewritten. Fixed little-endian layout allows code to be optimised and only added at compile time for big endian systems. 2.2 Support for pseudo files added. New Mksquashfs options Mksquashfs now supports pseudo files, these allow fake directories, character and block devices to be specified and added to the Squashfs filesystem being built, rather than requiring them to be present in the source directories. This, for example, allows device nodes to be added to the filesystem without requiring root access. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 05/17] Squashfs: symlink operations
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/symlink.c | 118 +
1 files changed, 118 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/symlink.c b/fs/squashfs/symlink.c
new file mode 100644
index 000..83d8788
--- /dev/null
+++ b/fs/squashfs/symlink.c
@@ -0,0 +1,118 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * symlink.c
+ */
+
+/*
+ * This file implements code to handle symbolic links.
+ *
+ * The data contents of symbolic links are stored inside the symbolic
+ * link inode within the inode table. This allows the normally small symbolic
+ * link to be compressed as part of the inode table, achieving much greater
+ * compression than if the symbolic link was compressed individually.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/kernel.h
+#include linux/slab.h
+#include linux/string.h
+#include linux/pagemap.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+static int squashfs_symlink_readpage(struct file *file, struct page *page)
+{
+ struct inode *inode = page-mapping-host;
+ struct super_block *sb = inode-i_sb;
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ int index = page-index PAGE_CACHE_SHIFT;
+ u64 block = squashfs_i(inode)-start;
+ int offset = squashfs_i(inode)-offset;
+ int length = min_t(int, i_size_read(inode) - index, PAGE_CACHE_SIZE);
+ int bytes, copied;
+ void *pageaddr;
+ struct squashfs_cache_entry *entry;
+
+ TRACE(Entered squashfs_symlink_readpage, page index %ld, start block
+ %llx, offset %x\n, page-index, block, offset);
+
+ /*
+* Skip index bytes into symlink metadata.
+*/
+ if (index) {
+ bytes = squashfs_read_metadata(sb, NULL, block, offset,
+ index);
+ if (bytes 0) {
+ ERROR(Unable to read symlink [%llx:%x]\n,
+ squashfs_i(inode)-start,
+ squashfs_i(inode)-offset);
+ goto error_out;
+ }
+ }
+
+ /*
+* Read length bytes from symlink metadata. Squashfs_read_metadata
+* is not used here because it can sleep and we want to use
+* kmap_atomic to map the page. Instead call the underlying
+* squashfs_cache_get routine. As length bytes may overlap metadata
+* blocks, we may need to call squashfs_cache_get multiple times.
+*/
+ for (bytes = 0; bytes length; offset = 0, bytes += copied) {
+ entry = squashfs_cache_get(sb, msblk-block_cache, block, 0);
+ if (entry-error) {
+ ERROR(Unable to read symlink [%llx:%x]\n,
+ squashfs_i(inode)-start,
+ squashfs_i(inode)-offset);
+ squashfs_cache_put(entry);
+ goto error_out;
+ }
+
+ pageaddr = kmap_atomic(page, KM_USER0);
+ copied = squashfs_copy_data(pageaddr + bytes, entry, offset,
+ length - bytes);
+ if (copied == length - bytes)
+ memset(pageaddr + length, 0, PAGE_CACHE_SIZE - length);
+ else
+ block = entry-next_index;
+ kunmap_atomic(pageaddr, KM_USER0);
+ squashfs_cache_put(entry);
+ }
+
+ flush_dcache_page(page);
+ SetPageUptodate(page);
+ unlock_page(page);
+ return 0;
+
+error_out:
+ SetPageError(page);
+ unlock_page(page);
+ return 0;
+}
+
+
+const struct address_space_operations squashfs_symlink_aops = {
+ .readpage = squashfs_symlink_readpage
+};
--
1.5.6.3
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to majord...@vger.kernel.org
More majordomo info at http
[PATCH V3 01/17] Squashfs: inode operations
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/inode.c | 346 +++
1 files changed, 346 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/inode.c b/fs/squashfs/inode.c
new file mode 100644
index 000..7a63398
--- /dev/null
+++ b/fs/squashfs/inode.c
@@ -0,0 +1,346 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * inode.c
+ */
+
+/*
+ * This file implements code to create and read inodes from disk.
+ *
+ * Inodes in Squashfs are identified by a 48-bit inode which encodes the
+ * location of the compressed metadata block containing the inode, and the byte
+ * offset into that block where the inode is placed (block, offset).
+ *
+ * To maximise compression there are different inodes for each file type
+ * (regular file, directory, device, etc.), the inode contents and length
+ * varying with the type.
+ *
+ * To further maximise compression, two types of regular file inode and
+ * directory inode are defined: inodes optimised for frequently occurring
+ * regular files and directories, and extended types where extra
+ * information has to be stored.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+/*
+ * Initialise VFS inode with the base inode information common to all
+ * Squashfs inode types. Sqsh_ino contains the unswapped base inode
+ * off disk.
+ */
+static int squashfs_new_inode(struct super_block *sb, struct inode *inode,
+ struct squashfs_base_inode *sqsh_ino)
+{
+ int err;
+
+ err = squashfs_get_id(sb, le16_to_cpu(sqsh_ino-uid), inode-i_uid);
+ if (err)
+ return err;
+
+ err = squashfs_get_id(sb, le16_to_cpu(sqsh_ino-guid), inode-i_gid);
+ if (err)
+ return err;
+
+ inode-i_ino = le32_to_cpu(sqsh_ino-inode_number);
+ inode-i_mtime.tv_sec = le32_to_cpu(sqsh_ino-mtime);
+ inode-i_atime.tv_sec = inode-i_mtime.tv_sec;
+ inode-i_ctime.tv_sec = inode-i_mtime.tv_sec;
+ inode-i_mode = le16_to_cpu(sqsh_ino-mode);
+ inode-i_size = 0;
+
+ return err;
+}
+
+
+struct inode *squashfs_iget(struct super_block *sb, long long ino,
+ unsigned int ino_number)
+{
+ struct inode *inode = iget_locked(sb, ino_number);
+ int err;
+
+ TRACE(Entered squashfs_iget\n);
+
+ if (!inode)
+ return ERR_PTR(-ENOMEM);
+ if (!(inode-i_state I_NEW))
+ return inode;
+
+ err = squashfs_read_inode(inode, ino);
+ if (err) {
+ iget_failed(inode);
+ return ERR_PTR(err);
+ }
+
+ unlock_new_inode(inode);
+ return inode;
+}
+
+
+/*
+ * Initialise VFS inode by reading inode from inode table (compressed
+ * metadata). The format and amount of data read depends on type.
+ */
+int squashfs_read_inode(struct inode *inode, long long ino)
+{
+ struct super_block *sb = inode-i_sb;
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ u64 block = SQUASHFS_INODE_BLK(ino) + msblk-inode_table;
+ int err, type, offset = SQUASHFS_INODE_OFFSET(ino);
+ union squashfs_inode squashfs_ino;
+ struct squashfs_base_inode *sqshb_ino = squashfs_ino.base;
+
+ TRACE(Entered squashfs_read_inode\n);
+
+ /*
+* Read inode base common to all inode types.
+*/
+ err = squashfs_read_metadata(sb, sqshb_ino, block,
+ offset, sizeof(*sqshb_ino));
+ if (err 0)
+ goto failed_read;
+
+ err = squashfs_new_inode(sb, inode, sqshb_ino);
+ if (err)
+ goto failed_read;
+
+ block = SQUASHFS_INODE_BLK(ino) + msblk-inode_table;
+ offset = SQUASHFS_INODE_OFFSET(ino);
+
+ type = le16_to_cpu(sqshb_ino-inode_type);
+ switch (type) {
+ case SQUASHFS_REG_TYPE: {
+ unsigned int frag_offset, frag_size, frag;
+ u64 frag_blk;
+ struct squashfs_reg_inode *sqsh_ino
[PATCH V3 17/17] MAINTAINERS: squashfs entry
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk --- MAINTAINERS |7 +++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/MAINTAINERS b/MAINTAINERS index befacf0..6ed506f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4049,6 +4049,13 @@ L: cbe-oss-...@ozlabs.org W: http://www.ibm.com/developerworks/power/cell/ S: Supported +SQUASHFS FILE SYSTEM +P: Phillip Lougher +M: phil...@lougher.demon.co.uk +L: squashfs-de...@lists.sourceforge.net (subscribers-only) +W: http://squashfs.org.uk +S: Maintained + SRM (Alpha) environment access P: Jan-Benedict Glaw M: jbg...@lug-owl.de -- 1.5.6.3 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 11/17] Squashfs: block operations
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/block.c | 274 +++
1 files changed, 274 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
new file mode 100644
index 000..c837dfc
--- /dev/null
+++ b/fs/squashfs/block.c
@@ -0,0 +1,274 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * block.c
+ */
+
+/*
+ * This file implements the low-level routines to read and decompress
+ * datablocks and metadata blocks.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/mutex.h
+#include linux/string.h
+#include linux/buffer_head.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+/*
+ * Read the metadata block length, this is stored in the first two
+ * bytes of the metadata block.
+ */
+static struct buffer_head *get_block_length(struct super_block *sb,
+ u64 *cur_index, int *offset, int *length)
+{
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ struct buffer_head *bh;
+
+ bh = sb_bread(sb, *cur_index);
+ if (bh == NULL)
+ return NULL;
+
+ if (msblk-devblksize - *offset == 1) {
+ *length = (unsigned char) bh-b_data[*offset];
+ put_bh(bh);
+ bh = sb_bread(sb, ++(*cur_index));
+ if (bh == NULL)
+ return NULL;
+ *length |= (unsigned char) bh-b_data[0] 8;
+ *offset = 1;
+ } else {
+ *length = (unsigned char) bh-b_data[*offset] |
+ (unsigned char) bh-b_data[*offset + 1] 8;
+ *offset += 2;
+ }
+
+ return bh;
+}
+
+
+/*
+ * Read and decompress a metadata block or datablock. Length is non-zero
+ * if a datablock is being read (the size is stored elsewhere in the
+ * filesystem), otherwise the length is obtained from the first two bytes of
+ * the metadata block. A bit in the length field indicates if the block
+ * is stored uncompressed in the filesystem (usually because compression
+ * generated a larger block - this does occasionally happen with zlib).
+ */
+int squashfs_read_data(struct super_block *sb, void **buffer, u64 index,
+ int length, u64 *next_index, int srclength)
+{
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ struct buffer_head **bh;
+ int offset = index ((1 msblk-devblksize_log2) - 1);
+ u64 cur_index = index msblk-devblksize_log2;
+ int bytes, compressed, b = 0, k = 0, page = 0, avail;
+
+
+ bh = kcalloc((msblk-block_size msblk-devblksize_log2) + 1,
+ sizeof(*bh), GFP_KERNEL);
+ if (bh == NULL)
+ return -ENOMEM;
+
+ if (length) {
+ /*
+* Datablock.
+*/
+ bytes = -offset;
+ compressed = SQUASHFS_COMPRESSED_BLOCK(length);
+ length = SQUASHFS_COMPRESSED_SIZE_BLOCK(length);
+ if (next_index)
+ *next_index = index + length;
+
+ TRACE(Block @ 0x%llx, %scompressed size %d, src size %d\n,
+ index, compressed ? : un, length, srclength);
+
+ if (length 0 || length srclength ||
+ (index + length) msblk-bytes_used)
+ goto read_failure;
+
+ for (b = 0; bytes length; b++, cur_index++) {
+ bh[b] = sb_getblk(sb, cur_index);
+ if (bh[b] == NULL)
+ goto block_release;
+ bytes += msblk-devblksize;
+ }
+ ll_rw_block(READ, b, bh);
+ } else {
+ /*
+* Metadata block.
+*/
+ if ((index + 2) msblk-bytes_used)
+ goto read_failure;
+
+ bh[0] = get_block_length(sb, cur_index, offset, length);
+ if (bh[0] == NULL
[PATCH V3 14/17] Squashfs: Kconfig entry
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk --- fs/Kconfig | 52 1 files changed, 52 insertions(+), 0 deletions(-) diff --git a/fs/Kconfig b/fs/Kconfig index ff0e819..2553e0b 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -894,6 +894,58 @@ config CRAMFS If unsure, say N. +config SQUASHFS + tristate SquashFS 4.0 - Squashed file system support + depends on BLOCK + select ZLIB_INFLATE + help + Saying Y here includes support for SquashFS 4.0 (a Compressed + Read-Only File System). Squashfs is a highly compressed read-only + filesystem for Linux. It uses zlib compression to compress both + files, inodes and directories. Inodes in the system are very small + and all blocks are packed to minimise data overhead. Block sizes + greater than 4K are supported up to a maximum of 1 Mbytes (default + block size 128K). SquashFS 4.0 supports 64 bit filesystems and files + (larger than 4GB), full uid/gid information, hard links and + timestamps. + + Squashfs is intended for general read-only filesystem use, for + archival use (i.e. in cases where a .tar.gz file may be used), and in + embedded systems where low overhead is needed. Further information + and tools are available from http://squashfs.sourceforge.net. + + If you want to compile this as a module ( = code which can be + inserted in and removed from the running kernel whenever you want), + say M here and read file:Documentation/modules.txt. The module + will be called squashfs. Note that the root file system (the one + containing the directory /) cannot be compiled as a module. + + If unsure, say N. + +config SQUASHFS_EMBEDDED + + bool Additional option for memory-constrained systems + depends on SQUASHFS + default n + help + Saying Y here allows you to specify cache size. + + If unsure, say N. + +config SQUASHFS_FRAGMENT_CACHE_SIZE + int Number of fragments cached if SQUASHFS_EMBEDDED + depends on SQUASHFS + default 3 + help + By default SquashFS caches the last 3 fragments read from + the filesystem. Increasing this amount may mean SquashFS + has to re-read fragments less often from disk, at the expense + of extra system memory. Decreasing this amount will mean + SquashFS uses less memory at the expense of extra reads from disk. + + Note there must be at least one cached fragment. Anything + much more than three will probably not make much difference. + config VXFS_FS tristate FreeVxFS file system support (VERITAS VxFS(TM) compatible) depends on BLOCK -- 1.5.6.3 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 02/17] Squashfs: directory lookup operations
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/namei.c | 242 +++
1 files changed, 242 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/namei.c b/fs/squashfs/namei.c
new file mode 100644
index 000..9e39865
--- /dev/null
+++ b/fs/squashfs/namei.c
@@ -0,0 +1,242 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * namei.c
+ */
+
+/*
+ * This file implements code to do filename lookup in directories.
+ *
+ * Like inodes, directories are packed into compressed metadata blocks, stored
+ * in a directory table. Directories are accessed using the start address of
+ * the metablock containing the directory and the offset into the
+ * decompressed block (block, offset).
+ *
+ * Directories are organised in a slightly complex way, and are not simply
+ * a list of file names. The organisation takes advantage of the
+ * fact that (in most cases) the inodes of the files will be in the same
+ * compressed metadata block, and therefore, can share the start block.
+ * Directories are therefore organised in a two level list, a directory
+ * header containing the shared start block value, and a sequence of directory
+ * entries, each of which share the shared start block. A new directory header
+ * is written once/if the inode start block changes. The directory
+ * header/directory entry list is repeated as many times as necessary.
+ *
+ * Directories are sorted, and can contain a directory index to speed up
+ * file lookup. Directory indexes store one entry per metablock, each entry
+ * storing the index/filename mapping to the first directory header
+ * in each metadata block. Directories are sorted in alphabetical order,
+ * and at lookup the index is scanned linearly looking for the first filename
+ * alphabetically larger than the filename being looked up. At this point the
+ * location of the metadata block the filename is in has been found.
+ * The general idea of the index is ensure only one metadata block needs to be
+ * decompressed to do a lookup irrespective of the length of the directory.
+ * This scheme has the advantage that it doesn't require extra memory overhead
+ * and doesn't require much extra storage on disk.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/string.h
+#include linux/dcache.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+/*
+ * Lookup name in the directory index, returning the location of the metadata
+ * block containing it, and the directory index this represents.
+ *
+ * If we get an error reading the index then return the part of the index
+ * (if any) we have managed to read - the index isn't essential, just
+ * quicker.
+ */
+static int get_dir_index_using_name(struct super_block *sb,
+ u64 *next_block, int *next_offset, u64 index_start,
+ int index_offset, int i_count, const char *name,
+ int len)
+{
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ int i, size, length = 0, err;
+ struct squashfs_dir_index *index;
+ char *str;
+
+ TRACE(Entered get_dir_index_using_name, i_count %d\n, i_count);
+
+ index = kmalloc(sizeof(*index) + SQUASHFS_NAME_LEN * 2 + 2, GFP_KERNEL);
+ if (index == NULL) {
+ ERROR(Failed to allocate squashfs_dir_index\n);
+ goto out;
+ }
+
+ str = index-name[SQUASHFS_NAME_LEN + 1];
+ strncpy(str, name, len);
+ str[len] = '\0';
+
+ for (i = 0; i i_count; i++) {
+ err = squashfs_read_metadata(sb, index, index_start,
+ index_offset, sizeof(*index));
+ if (err 0)
+ break;
+
+
+ size = le32_to_cpu(index-size) + 1;
+
+ err = squashfs_read_metadata(sb, index-name, index_start,
+ index_offset, size);
+ if (err 0)
+ break;
+
+ index-name[size] = '\0
[PATCH V3 09/17] Squashfs: uid/gid lookup operations
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/id.c | 94 ++
1 files changed, 94 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/id.c b/fs/squashfs/id.c
new file mode 100644
index 000..3795b83
--- /dev/null
+++ b/fs/squashfs/id.c
@@ -0,0 +1,94 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * id.c
+ */
+
+/*
+ * This file implements code to handle uids and gids.
+ *
+ * For space efficiency regular files store uid and gid indexes, which are
+ * converted to 32-bit uids/gids using an id look up table. This table is
+ * stored compressed into metadata blocks. A second index table is used to
+ * locate these. This second index table for speed of access (and because it
+ * is small) is read at mount time and cached in memory.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+/*
+ * Map uid/gid index into real 32-bit uid/gid using the id look up table
+ */
+int squashfs_get_id(struct super_block *sb, unsigned int index,
+ unsigned int *id)
+{
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ int block = SQUASHFS_ID_BLOCK(index);
+ int offset = SQUASHFS_ID_BLOCK_OFFSET(index);
+ u64 start_block = le64_to_cpu(msblk-id_table[block]);
+ __le32 disk_id;
+ int err;
+
+ err = squashfs_read_metadata(sb, disk_id, start_block, offset,
+ sizeof(disk_id));
+ if (err 0)
+ return err;
+
+ *id = le32_to_cpu(disk_id);
+ return 0;
+}
+
+
+/*
+ * Read uncompressed id lookup table indexes from disk into memory
+ */
+__le64 *squashfs_read_id_index_table(struct super_block *sb,
+ u64 id_table_start, unsigned short no_ids)
+{
+ unsigned int length = SQUASHFS_ID_BLOCK_BYTES(no_ids);
+ __le64 *id_table;
+ int err;
+
+ TRACE(In read_id_index_table, length %d\n, length);
+
+ /* Allocate id lookup table indexes */
+ id_table = kmalloc(length, GFP_KERNEL);
+ if (id_table == NULL) {
+ ERROR(Failed to allocate id index table\n);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ err = squashfs_read_table(sb, id_table, id_table_start, length);
+ if (err 0) {
+ ERROR(unable to read id index table\n);
+ kfree(id_table);
+ return ERR_PTR(err);
+ }
+
+ return id_table;
+}
--
1.5.6.3
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V3 10/17] Squashfs: cache operations
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/cache.c | 412 +++
1 files changed, 412 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/cache.c b/fs/squashfs/cache.c
new file mode 100644
index 000..f29eda1
--- /dev/null
+++ b/fs/squashfs/cache.c
@@ -0,0 +1,412 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * cache.c
+ */
+
+/*
+ * Blocks in Squashfs are compressed. To avoid repeatedly decompressing
+ * recently accessed data Squashfs uses two small metadata and fragment caches.
+ *
+ * This file implements a generic cache implementation used for both caches,
+ * plus functions layered ontop of the generic cache implementation to
+ * access the metadata and fragment caches.
+ *
+ * To avoid out of memory and fragmentation isssues with vmalloc the cache
+ * uses sequences of kmalloced PAGE_CACHE_SIZE buffers.
+ *
+ * It should be noted that the cache is not used for file datablocks, these
+ * are decompressed and cached in the page-cache in the normal way. The
+ * cache is only used to temporarily cache fragment and metadata blocks
+ * which have been read as as a result of a metadata (i.e. inode or
+ * directory) or fragment access. Because metadata and fragments are packed
+ * together into blocks (to gain greater compression) the read of a particular
+ * piece of metadata or fragment will retrieve other metadata/fragments which
+ * have been packed with it, these because of locality-of-reference may be read
+ * in the near future. Temporarily caching them ensures they are available for
+ * near future access without requiring an additional read and decompress.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/vmalloc.h
+#include linux/sched.h
+#include linux/spinlock.h
+#include linux/wait.h
+#include linux/zlib.h
+#include linux/pagemap.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+/*
+ * Look-up block in cache, and increment usage count. If not in cache, read
+ * and decompress it from disk.
+ */
+struct squashfs_cache_entry *squashfs_cache_get(struct super_block *sb,
+ struct squashfs_cache *cache, u64 block, int length)
+{
+ int i, n;
+ struct squashfs_cache_entry *entry;
+
+ spin_lock(cache-lock);
+
+ while (1) {
+ for (i = 0; i cache-entries; i++)
+ if (cache-entry[i].block == block)
+ break;
+
+ if (i == cache-entries) {
+ /*
+* Block not in cache, if all cache entries are used
+* go to sleep waiting for one to become available.
+*/
+ if (cache-unused == 0) {
+ cache-num_waiters++;
+ spin_unlock(cache-lock);
+ wait_event(cache-wait_queue, cache-unused);
+ spin_lock(cache-lock);
+ cache-num_waiters--;
+ continue;
+ }
+
+ /*
+* At least one unused cache entry. A simple
+* round-robin strategy is used to choose the entry to
+* be evicted from the cache.
+*/
+ i = cache-next_blk;
+ for (n = 0; n cache-entries; n++) {
+ if (cache-entry[i].refcount == 0)
+ break;
+ i = (i + 1) % cache-entries;
+ }
+
+ cache-next_blk = (i + 1) % cache-entries;
+ entry = cache-entry[i];
+
+ /*
+* Initialise choosen cache entry, and fill it in from
+* disk.
+*/
+ cache-unused--;
+ entry-block = block
[PATCH V3 12/17] Squashfs: header files
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk
---
fs/squashfs/squashfs.h | 90 ++
fs/squashfs/squashfs_fs.h| 381 ++
fs/squashfs/squashfs_fs_i.h | 45 +
fs/squashfs/squashfs_fs_sb.h | 76 +
4 files changed, 592 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/squashfs.h b/fs/squashfs/squashfs.h
new file mode 100644
index 000..6b2515d
--- /dev/null
+++ b/fs/squashfs/squashfs.h
@@ -0,0 +1,90 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * squashfs.h
+ */
+
+#define TRACE(s, args...) pr_debug(SQUASHFS: s, ## args)
+
+#define ERROR(s, args...) pr_err(SQUASHFS error: s, ## args)
+
+#define WARNING(s, args...)pr_warning(SQUASHFS: s, ## args)
+
+static inline struct squashfs_inode_info *squashfs_i(struct inode *inode)
+{
+ return list_entry(inode, struct squashfs_inode_info, vfs_inode);
+}
+
+/* block.c */
+extern int squashfs_read_data(struct super_block *, void **, u64, int, u64 *,
+ int);
+
+/* cache.c */
+extern struct squashfs_cache *squashfs_cache_init(char *, int, int);
+extern void squashfs_cache_delete(struct squashfs_cache *);
+extern struct squashfs_cache_entry *squashfs_cache_get(struct super_block *,
+ struct squashfs_cache *, u64, int);
+extern void squashfs_cache_put(struct squashfs_cache_entry *);
+extern int squashfs_copy_data(void *, struct squashfs_cache_entry *, int, int);
+extern int squashfs_read_metadata(struct super_block *, void *, u64 *,
+ int *, int);
+extern struct squashfs_cache_entry *squashfs_get_fragment(struct super_block *,
+ u64, int);
+extern struct squashfs_cache_entry *squashfs_get_datablock(struct super_block
*,
+ u64, int);
+extern int squashfs_read_table(struct super_block *, void *, u64, int);
+
+/* export.c */
+extern __le64 *squashfs_read_inode_lookup_table(struct super_block *, u64,
+ unsigned int);
+
+/* fragment.c */
+extern int squashfs_frag_lookup(struct super_block *, unsigned int, u64 *);
+extern __le64 *squashfs_read_fragment_index_table(struct super_block *,
+ u64, unsigned int);
+
+/* id.c */
+extern int squashfs_get_id(struct super_block *, unsigned int, unsigned int *);
+extern __le64 *squashfs_read_id_index_table(struct super_block *, u64,
+ unsigned short);
+
+/* inode.c */
+extern struct inode *squashfs_iget(struct super_block *, long long,
+ unsigned int);
+extern int squashfs_read_inode(struct inode *, long long);
+
+/*
+ * Inodes and files operations
+ */
+
+/* dir.c */
+extern const struct file_operations squashfs_dir_ops;
+
+/* export.c */
+extern const struct export_operations squashfs_export_ops;
+
+/* file.c */
+extern const struct address_space_operations squashfs_aops;
+
+/* namei.c */
+extern const struct inode_operations squashfs_dir_inode_ops;
+
+/* symlink.c */
+extern const struct address_space_operations squashfs_symlink_aops;
diff --git a/fs/squashfs/squashfs_fs.h b/fs/squashfs/squashfs_fs.h
new file mode 100644
index 000..6840da1
--- /dev/null
+++ b/fs/squashfs/squashfs_fs.h
@@ -0,0 +1,381 @@
+#ifndef SQUASHFS_FS
+#define SQUASHFS_FS
+/*
+ * Squashfs
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher phil...@lougher.demon.co.uk
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin
[PATCH V3 13/17] Squashfs: Makefiles
Signed-off-by: Phillip Lougher phil...@lougher.demon.co.uk --- fs/Makefile |1 + fs/squashfs/Makefile |8 2 files changed, 9 insertions(+), 0 deletions(-) diff --git a/fs/Makefile b/fs/Makefile index e6f423d..3f8843c 100644 --- a/fs/Makefile +++ b/fs/Makefile @@ -73,6 +73,7 @@ obj-$(CONFIG_JBD) += jbd/ obj-$(CONFIG_JBD2) += jbd2/ obj-$(CONFIG_EXT2_FS) += ext2/ obj-$(CONFIG_CRAMFS) += cramfs/ +obj-$(CONFIG_SQUASHFS) += squashfs/ obj-y += ramfs/ obj-$(CONFIG_HUGETLBFS)+= hugetlbfs/ obj-$(CONFIG_CODA_FS) += coda/ diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile new file mode 100644 index 000..8258cf9 --- /dev/null +++ b/fs/squashfs/Makefile @@ -0,0 +1,8 @@ +# +# Makefile for the linux squashfs routines. +# + +obj-$(CONFIG_SQUASHFS) += squashfs.o +squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o +squashfs-y += namei.o super.o symlink.o +#squashfs-y += squashfs2_0.o -- 1.5.6.3 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: LZMA inclusion
Lasse Collin wrote: Phillip Lougher wrote: Currently, as mentioned above, Squashfs decompresses into a single contiguous output buffer. But, due to the linux kernel mailing list's dislike of vmalloc, this is being changed. In future Squashfs will decompress into a sequence of 4 KiB output buffers (possibly in the page cache). To my understanding, using 4 KiB output buffers can make sense only if the dictionary size is significantly smaller than the Squashfs block size. This is because an output buffer scattered to 4 KiB pieces means that the the dictionary has to be vmalloced as part of the LZMA decoder state. For example, if the dictionary size is equal to the Squashfs block size, the same amount of memory that earlier Squashfs versions vmalloced for the output buffer is now vmalloced by the uncompression code for the dictionary. Plus, memcpy is needed to get the data from the dictionary to the 4 KiB output buffers. The issue that moving to 4 KiB output buffers solves is it reduces significantly the number of 1 MiB (or 128 KiB for the default block size) buffers that need to be vmalloced. Squashfs caches the last 3 fragment buffers read, and moving to 4 KiB buffers eliminates these vmallocs. Obviously moving to 4 KiB output buffers will require a 1 MiB dictionary workspace to be vmalloced, but this is still much less than the 3 buffers that currently need to be vmalloced. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: LZMA inclusion
Jörn Engel wrote: On Sun, 7 December 2008 23:32:32 +, Phillip Lougher wrote: Currently, as mentioned above, Squashfs decompresses into a single contiguous output buffer. But, due to the linux kernel mailing list's dislike of vmalloc, this is being changed. Don't blame lkml, blame Intel and IBM. Back in the days of the 386, a beefy machine had 8MB of physical memory and 4GB of virtual memory space. Noone had to worry about fragmentation anymore. If you needed a 1MB buffer, you'd just round up some 256 pages and instruct the mmu to map them into a large contiguous address range in the virtual address space. Life was good indeed. But physical memory has constantly grown since, while the virtual memory space has for a long time stagnated. Intel even introduced some hardware hacks to use up to 64GB of physical memory with a measly 4GB of virtual memory. Now it was _virtual_ memory fragmentation that you had to worry about. These days most CPUs you'd buy are 64bit, so virtual memory space has become useful again. But as a kernel hacker, you have little control over what hardware everyone is using. And those weird systems with more physical than virtual memory are still around. :( Jörn Yes, I'm aware of the issues with vmalloc on older hardware. -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: LZMA inclusion
Jörn Engel wrote: On Sat, 6 December 2008 23:56:50 +0200, Lasse Collin wrote: Since you are improving the crypto API, maybe it would be a good idea to add a flag to tell the decoder that the whole output buffer will be kept available to the multi-call decoder. I'm not convinced this is the right direction. One of the constraints of kernel programming is that large contiguous are hard to come by. The mm subsystem makes no guarantees that you will be able to allocate 1MiB or contiguous memory. On a 32bit system with highmem, it may even become hard to get 1MiB from vmalloc. This is an important issue, on the last Squashfs submission attempt, its use of vmalloc to allocate up to 1MiB contiguous blocks for decompression was brought up. Any LZMA implementation which requires 1MiB vmalloced input and output buffers will probably face similar problems. So another approach would be to ignore the one-shot debate and concentrate on taking a pagevec instead of a buffer (as in a void * pointer). That would certainly be useful for other compressed filesystems and without checking the code (I forgot where the squashfs git tree was) I claim it should be useful for squashfs as well. Squashfs doesn't use one-shot decoding with zlib for performance and memory issues. Input data is split across buffer_heads (4 KiB or less per buffer_head), and calling zlib repeatedly for each separate buffer_head eliminates the necessary memcpy into a larger input buffer, eliminates the memory overhead for this buffer, and ensures only the first buffer_head needs to be waited on (for arrival off disk) before decompression starts. Currently, as mentioned above, Squashfs decompresses into a single contiguous output buffer. But, due to the linux kernel mailing list's dislike of vmalloc, this is being changed. In future Squashfs will decompress into a sequence of 4 KiB output buffers (possibly in the page cache). One-shot LZMA decoding therefore isn't going to work very well with future versions of Squashfs, obviously a solution (as is currently done with the Squashfs-LZMA patches) is to use separately allocated contiguous input/output buffers, and memcpy into and out of them, but this isn't particularly ideal. The discussion about using the output buffer as the temporary workspace (as it isn't touched until after decompression is completely finished) will work with the current version of Squashfs, but it isn't going to work with later versions unless the LZMA code can be changed to work with a list of discontiguous output buffers (i.e. a scatter-gather type list). So it looks inevitable that a separately vmalloced workspace buffer will be required. Phillip Jörn -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V2 10/16] Squashfs: cache operations
Jörn Engel wrote: Only one of your problems seems to be real. Not sure if or how we can solve that one, though. Sorry don't agree. But I'm not going to argue this like a couple of old maids. I'll keep what I currently do unless Andrew Morton or someone else says it's not getting mainlined unless it's changed. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V2 00/16] Squashfs: compressed read-only filesystem
Matt Mackall wrote: On Wed, 2008-10-29 at 01:49 +, Phillip Lougher wrote: Hi, This a respin of the Squashfs patches incorporating the review comments received. Thanks to everyone who have sent comments. I read over the v3 source a few weeks ago and must say this looks greatly improved. Yes thanks. The v3 source was a bit of a mess, it had grown organically from the earliest version of Squashfs, and long needed restructuring, closer attention to coding standards, and commenting. I think the v4 source is a major improvement, it's partially thanks to CELF that I got the necessary time off work to knock it into shape. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH V2 10/16] Squashfs: cache operations
Jörn Engel wrote: On Wed, 29 October 2008 01:49:56 +, Phillip Lougher wrote: +/* + * Blocks in Squashfs are compressed. To avoid repeatedly decompressing + * recently accessed data Squashfs uses two small metadata and fragment caches. + * + * This file implements a generic cache implementation used for both caches, + * plus functions layered ontop of the generic cache implementation to + * access the metadata and fragment caches. + */ I tend to agree with Andrew that a lot of this should be done by the page cache instead. One of the problems seems to be that your blocksize can exceed page size and there really isn't any infrastructure to deal with such cases yet. Bufferheads deal with blocks smaller than a page, not the other way around. I thought about using the page cache, but, the fact that blocksizes exceed the page cache size is only one of a number of reasons why I prefer the current solution, there is also simplicity and speed to consider. There are three types of compressed block in Squashfs: datablocks, fragments, and metadata blocks. Of these datablocks (by far the largest number of blocks) are decompressed and pushed into the page cache, and are not otherwise cached by Squashfs. This, obviously (?), is because they contain data for only one file, and so at time of access there is a readily available address space to push the data into. Metadata and fragment blocks are different in that when accessed and decompressed (say for an inode or for a particular tail-end block) they will contain metadata or tail-ends for other files. This data could be thrown away but due to locality of reference it makes sense to temporarily cache it in-case a near future file access references the data. But it doesn't make much sense to cache it more than temporarily, much of the data will probably not be reused, and it exists compressed in the buffer cache. The squashfs cache is therefore designed to cache only the last couple of metadata and fragment blocks. It is also designed to be simple and extremely fast. The maximum size of the metadata cache is only 64 KiB. Simplicity and speed is extremely important. The squashfs_metadata_read() wrapper around the cache is designed to step through the metadata a structure at a time (20 or so bytes), updating the read position in the metadata each call, with more metadata cache blocks being read and decompressed as necessary. The common case where the metadata is already in the cache (because we're stepping through it 20 or so bytes at a time), is designed to be extremely fast - a spinlock and array search only. I recently optimised the cache to use spinlocks rather than mutexes and reduced the lock holding time (necessary to move to spinlocks), and this resulted in a 20%+ speed improvement in reading squashfs filesystems. Given the above using an address space in the page cache will result in greater complexity, more memory overhead, and much slower operation. There's a number of reasons for this. 1. The amount and life-span of the data stored in the page cache is outside of Squashfs' control. As explained above it only makes sense to temporarily cache the last couple of metadata and fragment blocks. Using the page cache (if a 'large' address space is used) for these keeps more of them around for longer than necessary, and will potentially cause more worthy datablocks to be flushed on memory pressure. 2. The address space will be caching uncompressed data, the squashfs references to this data are the compressed locations within the filesystem. There doesn't exist a one-to-one linear mapping from compressed location to decompressed location in the address space. This means a lookup table still needs to be used to store the mapping from compressed location to decompressed location in the address space. Now this lookup table (however implemented) is itself at least as complex as my current cache implementation. 3. Once the locations of the decompressed pages in the address space have been found, they'll need to be looked up in the page cache, and this has to be done for every 4K page. With the default fragment size of 128 KiB this means 32 separate lookups. Somewhat slower than one spinlock and array search per 128 KiB block in the squashfs cache implementation. Comments, especially those of the form you've got this completely wrong, and you can use the page cache like this, which will be simpler and faster than your current implementation welcome :) I'm not adverse to using the page cache, but I can't see how it will be simpler or faster than the current implementation. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V2 00/16] Squashfs: compressed read-only filesystem
Hi, This a respin of the Squashfs patches incorporating the review comments received. Thanks to everyone who have sent comments. Summary of changes in patch respin: 1. Functions changed to return 0 on success and -ESOMETHING on error 2. Header files moved from include/linux to fs/squashfs 3. Variables changed to use sb and inode 4. Number of squashfs_read_metadata() parameters reduced 5. Xattr placeholder code tweaked 6. TRACE and ERROR macros fixed to use pr_debug and pr_warning 7. Some obsolete macros in squashfs_fs.h removed 8. A number of gotos to return statements replaced with direct returns 9. Sparse with endian checking (make C=2 CHECKFLAGS=-D__CHECK_ENDIAN__) errors fixed 10. get_dir_index_using_name() misaligned access fixed 11. Fix a couple of printk warnings on PPC64 12. Shorten a number of variable names There is now a public git repository on kernel.org. Pull/clone from git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-2.6.git These 16 patches are against 2.6.28-rc2. Following is the original re-submission oveview, detailing the major changes since the original 2005 submission, and a case for its inclusion. Thanks Phillip This is a second attempt at mainlining Squashfs. The first attempt was way way back in early 2005 :-) Since then the filesystem layout has undergone two major revisions, and the kernel code has almost been completely rewritten. Both of these were to address the criticisms made at the original attempt. Summary of changes: 1. Filesystem layout is now 64-bit, in theory filesystems and files can be 2^64 in size. 2. Filesystem is now fixed little-endian. 3. . and .. are now returned by readdir. 4. Sparse files are now supported. 5. Filesystem is now exportable (NFS etc.). 6. Datablocks up to 1 Mbyte are now supported. Codewise all of the packed bit-fields and the swap macros have been removed in favour of aligned structures and in-line swapping using leXX_to_cpu(). The code has also been extensively restructured, reformatted to kernel coding standards and commented. Previously there was resistance to the inclusion of another compressed filesystem when Linux already had cramfs. There was pressure for a strong case to be made for the inclusion of Squashfs. Hopefully the case for the inclusion of other compressed filesystems has now already been answered over the last couple of years, however, it is worth listing the features of Squashfs over cramfs, which is still the only read-only compressed filesystem in mainline. Max filesystem size: cramfs 16 Mbytes, Squashfs 64-bit filesystem Max filesize: cramfs 16 Mbytes, Squashfs 64-bit filesystem Block size: cramfs 4K, Squashfs default 128K, max 1Mbyte Tail-end packing: cramfs no, Squashfs yes Directory indexes: cramfs no, Squashfs yes Compressed metadata: cramfs no, Squashfs yes Hard link support: cramfs no, Squashfs yes Support for . and .. in readdir: cramfs no, Squashfs yes Real inode numbers: cramfs no, Squashfs yes. Cramfs gives device inodes, fifo and empty directories the same inode of 1! Exportable filesystem (NFS, etc.): cramfs no, Squashfs yes Active maintenance: cramfs no (it is listed as orphaned, probably no active work for years), Squashfs yes Sorry for the list formatting, but many email readers are very unforgiving displaying tabbed lists and so I avoided them. For those that want hard performance statistics http://tree.celinuxforum.org/CelfPubWiki/SquashFsComparisons gives a full comparison of the performance of Squashfs against cramfs, zisofs, cloop and ext3. I made these tests a number of years ago using Squashfs 2.1, but they are still valid. In fact the performance should now be better. Cramfs is a limited filesystem, it's good for some embedded users but not now much else, its layout and features hasn't changed in the eight years+ since its release. Squashfs, despite never being in mainline, has been actively developed for over six years, and in that time has gone through four layout revisions, each revision improving compression and performance where limitations were found. For an often dismissed filesystem, Squashfs has advanced features such as metadata compression and tail-end packing for greater compression, and directory indexes for faster dentry operations. Despite not being in mainline, it is widely used. It is packaged by all major distributions (Ubuntu, Fedora, Debian, SUSE, Gentoo), it is used on most LiveCDs, it is extensively used in embedded systems (STBs, routers, mobile phones), and notably is used in such things as Splashtop and the Amazon Kindle. Anyway that's my case for inclusion. If any readers want Squashfs mainlined it's probably now a good time to offer support! There are 16 patches in the patch set, and the patches are against the latest linux-next tree (linux 2.6.27-next-20081016). Finally, I would like to acknowledge the financial support of the Consumer Embedded
[PATCH V2 16/16] MAINTAINERS: squashfs entry
Signed-off-by: Phillip Lougher [EMAIL PROTECTED] --- MAINTAINERS |7 +++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/MAINTAINERS b/MAINTAINERS index 16202c8..bc5b063 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3951,6 +3951,13 @@ L: [EMAIL PROTECTED] W: http://www.ibm.com/developerworks/power/cell/ S: Supported +SQUASHFS FILE SYSTEM +P: Phillip Lougher +M: [EMAIL PROTECTED] +L: [EMAIL PROTECTED] (subscribers-only) +W: http://squashfs.org.uk +S: Maintained + SRM (Alpha) environment access P: Jan-Benedict Glaw M: [EMAIL PROTECTED] -- 1.5.2.5 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V2 15/16] Squashfs: initrd support
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
init/do_mounts_rd.c | 14 ++
1 files changed, 14 insertions(+), 0 deletions(-)
diff --git a/init/do_mounts_rd.c b/init/do_mounts_rd.c
index a7c748f..0f0f0cf 100644
--- a/init/do_mounts_rd.c
+++ b/init/do_mounts_rd.c
@@ -9,6 +9,7 @@
#include linux/string.h
#include do_mounts.h
+#include ../fs/squashfs/squashfs_fs.h
int __initdata rd_prompt = 1;/* 1 = prompt for RAM disk, 0 = don't prompt */
@@ -41,6 +42,7 @@ static int __init crd_load(int in_fd, int out_fd);
* ext2
* romfs
* cramfs
+ * squashfs
* gzip
*/
static int __init
@@ -51,6 +53,7 @@ identify_ramdisk_image(int fd, int start_block)
struct ext2_super_block *ext2sb;
struct romfs_super_block *romfsb;
struct cramfs_super *cramfsb;
+ struct squashfs_super_block *squashfsb;
int nblocks = -1;
unsigned char *buf;
@@ -62,6 +65,7 @@ identify_ramdisk_image(int fd, int start_block)
ext2sb = (struct ext2_super_block *) buf;
romfsb = (struct romfs_super_block *) buf;
cramfsb = (struct cramfs_super *) buf;
+ squashfsb = (struct squashfs_super_block *) buf;
memset(buf, 0xe5, size);
/*
@@ -99,6 +103,16 @@ identify_ramdisk_image(int fd, int start_block)
goto done;
}
+ /* squashfs is at block zero too */
+ if (le32_to_cpu(squashfsb-s_magic) == SQUASHFS_MAGIC) {
+ printk(KERN_NOTICE
+ RAMDISK: squashfs filesystem found at block %d\n,
+ start_block);
+ nblocks = (le64_to_cpu(squashfsb-bytes_used) + BLOCK_SIZE - 1)
+ BLOCK_SIZE_BITS;
+ goto done;
+ }
+
/*
* Read block 1 to test for minix and ext2 superblock
*/
--
1.5.2.5
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V2 02/16] Squashfs: directory lookup operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/namei.c | 242 +++
1 files changed, 242 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/namei.c b/fs/squashfs/namei.c
new file mode 100644
index 000..db03ffe
--- /dev/null
+++ b/fs/squashfs/namei.c
@@ -0,0 +1,242 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * namei.c
+ */
+
+/*
+ * This file implements code to do filename lookup in directories.
+ *
+ * Like inodes, directories are packed into compressed metadata blocks, stored
+ * in a directory table. Directories are accessed using the start address of
+ * the metablock containing the directory and the offset into the
+ * decompressed block (block, offset).
+ *
+ * Directories are organised in a slightly complex way, and are not simply
+ * a list of file names. The organisation takes advantage of the
+ * fact that (in most cases) the inodes of the files will be in the same
+ * compressed metadata block, and therefore, can share the start block.
+ * Directories are therefore organised in a two level list, a directory
+ * header containing the shared start block value, and a sequence of directory
+ * entries, each of which share the shared start block. A new directory header
+ * is written once/if the inode start block changes. The directory
+ * header/directory entry list is repeated as many times as necessary.
+ *
+ * Directories are sorted, and can contain a directory index to speed up
+ * file lookup. Directory indexes store one entry per metablock, each entry
+ * storing the index/filename mapping to the first directory header
+ * in each metadata block. Directories are sorted in alphabetical order,
+ * and at lookup the index is scanned linearly looking for the first filename
+ * alphabetically larger than the filename being looked up. At this point the
+ * location of the metadata block the filename is in has been found.
+ * The general idea of the index is ensure only one metadata block needs to be
+ * decompressed to do a lookup irrespective of the length of the directory.
+ * This scheme has the advantage that it doesn't require extra memory overhead
+ * and doesn't require much extra storage on disk.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/string.h
+#include linux/dcache.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+/*
+ * Lookup name in the directory index, returning the location of the metadata
+ * block containing it, and the directory index this represents.
+ *
+ * If we get an error reading the index then return the part of the index
+ * (if any) we have managed to read - the index isn't essential, just
+ * quicker.
+ */
+static int get_dir_index_using_name(struct super_block *sb,
+ long long *next_block, int *next_offset,
+ long long index_start, int index_offset,
+ int i_count, const char *name, int len)
+{
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ int i, size, length = 0, err;
+ struct squashfs_dir_index *index;
+ char *str;
+
+ TRACE(Entered get_dir_index_using_name, i_count %d\n, i_count);
+
+ index = kmalloc(sizeof(*index) + SQUASHFS_NAME_LEN * 2 + 2, GFP_KERNEL);
+ if (index == NULL) {
+ ERROR(Failed to allocate squashfs_dir_index\n);
+ goto out;
+ }
+
+ str = index-name[SQUASHFS_NAME_LEN + 1];
+ strncpy(str, name, len);
+ str[len] = '\0';
+
+ for (i = 0; i i_count; i++) {
+ err = squashfs_read_metadata(sb, index, index_start,
+ index_offset, sizeof(*index));
+ if (err 0)
+ break;
+
+
+ size = le32_to_cpu(index-size) + 1;
+
+ err = squashfs_read_metadata(sb, index-name, index_start,
+ index_offset, size);
+ if (err 0)
+ break;
+
+ index-name[size] = '\0
[PATCH V2 11/16] Squashfs: block operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/block.c | 256 +++
1 files changed, 256 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
new file mode 100644
index 000..da5f88b
--- /dev/null
+++ b/fs/squashfs/block.c
@@ -0,0 +1,256 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * block.c
+ */
+
+/*
+ * This file implements the low-level routines to read and decompress
+ * datablocks and metadata blocks.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/mutex.h
+#include linux/string.h
+#include linux/buffer_head.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+/*
+ * Read the metadata block length, this is stored in the first two
+ * bytes of the metadata block.
+ */
+static struct buffer_head *get_block_length(struct super_block *sb,
+ long long *cur_index, int *offset, int *length)
+{
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ struct buffer_head *bh;
+
+ bh = sb_bread(sb, *cur_index);
+ if (bh == NULL)
+ return NULL;
+
+ if (msblk-devblksize - *offset == 1) {
+ *length = (unsigned char) bh-b_data[*offset];
+ brelse(bh);
+ bh = sb_bread(sb, ++(*cur_index));
+ if (bh == NULL)
+ return NULL;
+ *length |= (unsigned char) bh-b_data[0] 8;
+ *offset = 1;
+ } else {
+ *length = (unsigned char) bh-b_data[*offset] |
+ (unsigned char) bh-b_data[*offset + 1] 8;
+ *offset += 2;
+ }
+
+ return bh;
+}
+
+
+/*
+ * Read and decompress a metadata block or datablock. Length is non-zero
+ * if a datablock is being read (the size is stored elsewhere in the
+ * filesystem), otherwise the length is obtained from the first two bytes of
+ * the metadata block. A bit in the length field indicates if the block
+ * is stored uncompressed in the filesystem (usually because compression
+ * generated a larger block - this does occasionally happen with zlib).
+ */
+int squashfs_read_data(struct super_block *sb, void *buffer,
+ long long index, int length, long long *next_index,
+ int srclength)
+{
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ struct buffer_head **bh;
+ int offset = index ((1 msblk-devblksize_log2) - 1);
+ long long cur_index = index msblk-devblksize_log2;
+ int avail, bytes, compressed, b = 0, k = 0;
+ int c_byte = length;
+
+ bh = kcalloc((msblk-block_size msblk-devblksize_log2) + 1,
+ sizeof(*bh), GFP_KERNEL);
+ if (bh == NULL)
+ return -ENOMEM;
+
+ if (c_byte) {
+ /*
+* Datablock.
+*/
+ bytes = -offset;
+ compressed = SQUASHFS_COMPRESSED_BLOCK(c_byte);
+ c_byte = SQUASHFS_COMPRESSED_SIZE_BLOCK(c_byte);
+
+ TRACE(Block @ 0x%llx, %scompressed size %d, src size %d\n,
+ index, compressed ? : un, c_byte, srclength);
+
+ if (c_byte 0 || c_byte srclength || index 0 ||
+ (index + c_byte) msblk-bytes_used)
+ goto read_failure;
+
+ for (b = 0; bytes c_byte; b++, cur_index++) {
+ bh[b] = sb_getblk(sb, cur_index);
+ if (bh[b] == NULL)
+ goto block_release;
+ bytes += msblk-devblksize;
+ }
+ ll_rw_block(READ, b, bh);
+ } else {
+ /*
+* Metadata block.
+*/
+ if (index 0 || (index + 2) msblk-bytes_used)
+ goto read_failure;
+
+ bh[0] = get_block_length(sb, cur_index, offset, c_byte);
+ if (bh[0] == NULL)
+ goto read_failure
[PATCH V2 10/16] Squashfs: cache operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/cache.c | 315 +++
1 files changed, 315 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/cache.c b/fs/squashfs/cache.c
new file mode 100644
index 000..d121f31
--- /dev/null
+++ b/fs/squashfs/cache.c
@@ -0,0 +1,315 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * cache.c
+ */
+
+/*
+ * Blocks in Squashfs are compressed. To avoid repeatedly decompressing
+ * recently accessed data Squashfs uses two small metadata and fragment caches.
+ *
+ * This file implements a generic cache implementation used for both caches,
+ * plus functions layered ontop of the generic cache implementation to
+ * access the metadata and fragment caches.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/vmalloc.h
+#include linux/sched.h
+#include linux/spinlock.h
+#include linux/wait.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+/*
+ * Look-up block in cache, and increment usage count. If not in cache, read
+ * and decompress it from disk.
+ */
+struct squashfs_cache_entry *squashfs_cache_get(struct super_block *sb,
+ struct squashfs_cache *cache, long long block, int length)
+{
+ int i, n;
+ struct squashfs_cache_entry *entry;
+
+ spin_lock(cache-lock);
+
+ while (1) {
+ for (i = 0; i cache-entries; i++)
+ if (cache-entry[i].block == block)
+ break;
+
+ if (i == cache-entries) {
+ /*
+* Block not in cache, if all cache entries are locked
+* go to sleep waiting for one to become available.
+*/
+ if (cache-unused == 0) {
+ cache-waiting++;
+ spin_unlock(cache-lock);
+ wait_event(cache-wait_queue, cache-unused);
+ spin_lock(cache-lock);
+ cache-waiting--;
+ continue;
+ }
+
+ /*
+* At least one unlocked cache entry. A simple
+* round-robin strategy is used to choose the entry to
+* be evicted from the cache.
+*/
+ i = cache-next_blk;
+ for (n = 0; n cache-entries; n++) {
+ if (cache-entry[i].locked == 0)
+ break;
+ i = (i + 1) % cache-entries;
+ }
+
+ cache-next_blk = (i + 1) % cache-entries;
+ entry = cache-entry[i];
+
+ /*
+* Initialise choosen cache entry, and fill it in from
+* disk.
+*/
+ cache-unused--;
+ entry-block = block;
+ entry-locked = 1;
+ entry-pending = 1;
+ entry-waiting = 0;
+ entry-error = 0;
+ spin_unlock(cache-lock);
+
+ entry-length = squashfs_read_data(sb, entry-data,
+ block, length, entry-next_index,
+ cache-block_size);
+
+ spin_lock(cache-lock);
+
+ if (entry-length 0)
+ entry-error = entry-length;
+
+ entry-pending = 0;
+ spin_unlock(cache-lock);
+
+ /*
+* While filling this entry one or more other processes
+* have looked it up in the cache, and have slept
+* waiting for it to become available.
+*/
+ if (entry-waiting
[PATCH V2 12/16] Squashfs: header files
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/squashfs.h | 89 ++
fs/squashfs/squashfs_fs.h| 380 ++
fs/squashfs/squashfs_fs_i.h | 45 +
fs/squashfs/squashfs_fs_sb.h | 76 +
4 files changed, 590 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/squashfs.h b/fs/squashfs/squashfs.h
new file mode 100644
index 000..b852934
--- /dev/null
+++ b/fs/squashfs/squashfs.h
@@ -0,0 +1,89 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * squashfs.h
+ */
+
+#define TRACE(s, args...) pr_debug(SQUASHFS: s, ## args)
+
+#define ERROR(s, args...) pr_err(SQUASHFS error: s, ## args)
+
+#define WARNING(s, args...)pr_warning(SQUASHFS: s, ## args)
+
+static inline struct squashfs_inode_info *SQUASHFS_I(struct inode *inode)
+{
+ return list_entry(inode, struct squashfs_inode_info, vfs_inode);
+}
+
+/* block.c */
+extern int squashfs_read_data(struct super_block *, void *,
+ long long, int, long long *, int);
+
+/* cache.c */
+extern struct squashfs_cache *squashfs_cache_init(char *, int, int, int);
+extern void squashfs_cache_delete(struct squashfs_cache *);
+struct squashfs_cache_entry *squashfs_cache_get(struct super_block *,
+ struct squashfs_cache *, long long, int);
+void squashfs_cache_put(struct squashfs_cache *, struct squashfs_cache_entry
*);
+extern int squashfs_read_metadata(struct super_block *, void *,
+ long long *, int *, int);
+extern struct squashfs_cache_entry *get_cached_fragment(struct super_block *,
+ long long, int);
+extern void release_cached_fragment(struct squashfs_sb_info *,
+ struct squashfs_cache_entry *);
+
+/* export.c */
+extern __le64 *read_inode_lookup_table(struct super_block *, long long,
+ unsigned int);
+
+/* fragment.c */
+extern int get_fragment_location(struct super_block *, unsigned int,
+ long long *);
+extern __le64 *read_fragment_index_table(struct super_block *, long long,
+ unsigned int);
+
+/* id.c */
+extern int squashfs_get_id(struct super_block *, unsigned int, unsigned int *);
+extern __le64 *read_id_index_table(struct super_block *, long long,
+ unsigned short);
+
+/* inode.c */
+extern struct inode *squashfs_iget(struct super_block *, long long,
+ unsigned int);
+extern int squashfs_read_inode(struct inode *, long long);
+
+/*
+ * Inodes and files operations
+ */
+
+/* dir.c */
+extern const struct file_operations squashfs_dir_ops;
+
+/* export.c */
+extern const struct export_operations squashfs_export_ops;
+
+/* file.c */
+extern const struct address_space_operations squashfs_aops;
+
+/* namei.c */
+extern const struct inode_operations squashfs_dir_inode_ops;
+
+/* symlink.c */
+extern const struct address_space_operations squashfs_symlink_aops;
diff --git a/fs/squashfs/squashfs_fs.h b/fs/squashfs/squashfs_fs.h
new file mode 100644
index 000..eef85b1
--- /dev/null
+++ b/fs/squashfs/squashfs_fs.h
@@ -0,0 +1,380 @@
+#ifndef SQUASHFS_FS
+#define SQUASHFS_FS
+/*
+ * Squashfs
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * squashfs_fs.h
+ */
+
+#define SQUASHFS_CACHED_FRAGMENTS
[PATCH V2 09/16] Squashfs: uid/gid lookup operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/id.c | 89 ++
1 files changed, 89 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/id.c b/fs/squashfs/id.c
new file mode 100644
index 000..720973f
--- /dev/null
+++ b/fs/squashfs/id.c
@@ -0,0 +1,89 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * id.c
+ */
+
+/*
+ * This file implements code to handle uids and gids.
+ *
+ * For space efficiency regular files store uid and gid indexes, which are
+ * converted to 32-bit uids/gids using an id look up table. This table is
+ * stored compressed into metadata blocks. A second index table is used to
+ * locate these. This second index table for speed of access (and because it
+ * is small) is read at mount time and cached in memory.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+int squashfs_get_id(struct super_block *sb, unsigned int index,
+ unsigned int *id)
+{
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ int block = SQUASHFS_ID_BLOCK(index);
+ int offset = SQUASHFS_ID_BLOCK_OFFSET(index);
+ long long start_block = le64_to_cpu(msblk-id_table[block]);
+ __le32 disk_id;
+ int err;
+
+ err = squashfs_read_metadata(sb, disk_id, start_block, offset,
+ sizeof(disk_id));
+ if (err 0)
+ return err;
+
+ *id = le32_to_cpu(disk_id);
+ return 0;
+}
+
+
+__le64 *read_id_index_table(struct super_block *sb, long long id_table_start,
+ unsigned short no_ids)
+{
+ unsigned int length = SQUASHFS_ID_BLOCK_BYTES(no_ids);
+ __le64 *id_table;
+ int err;
+
+ TRACE(In read_id_index_table, length %d\n, length);
+
+ /* Allocate id index table */
+ id_table = kmalloc(length, GFP_KERNEL);
+ if (id_table == NULL) {
+ ERROR(Failed to allocate id index table\n);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ err = squashfs_read_data(sb, id_table, id_table_start, length |
+ SQUASHFS_COMPRESSED_BIT_BLOCK, NULL, length);
+ if (err 0) {
+ ERROR(unable to read id index table\n);
+ kfree(id_table);
+ return ERR_PTR(err);
+ }
+
+ return id_table;
+}
--
1.5.2.5
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V2 01/16] Squashfs: inode operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/inode.c | 346 +++
1 files changed, 346 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/inode.c b/fs/squashfs/inode.c
new file mode 100644
index 000..840c82b
--- /dev/null
+++ b/fs/squashfs/inode.c
@@ -0,0 +1,346 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * inode.c
+ */
+
+/*
+ * This file implements code to create and read inodes from disk.
+ *
+ * Inodes in Squashfs are identified by a 48-bit inode which encodes the
+ * location of the compressed metadata block containing the inode, and the byte
+ * offset into that block where the inode is placed (block, offset).
+ *
+ * To maximise compression there are different inodes for each file type
+ * (regular file, directory, device, etc.), the inode contents and length
+ * varying with the type.
+ *
+ * To further maximise compression, two types of regular file inode and
+ * directory inode are defined: inodes optimised for frequently occurring
+ * regular files and directories, and extended types where extra
+ * information has to be stored.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+/*
+ * Initialise VFS inode with the base inode information common to all
+ * Squashfs inode types. Sqsh_ino contains the unswapped base inode
+ * off disk.
+ */
+static int squashfs_new_inode(struct super_block *sb, struct inode *inode,
+ struct squashfs_base_inode *sqsh_ino)
+{
+ int err;
+
+ err = squashfs_get_id(sb, le16_to_cpu(sqsh_ino-uid), inode-i_uid);
+ if (err)
+ return err;
+
+ err = squashfs_get_id(sb, le16_to_cpu(sqsh_ino-guid), inode-i_gid);
+ if (err)
+ return err;
+
+ inode-i_ino = le32_to_cpu(sqsh_ino-inode_number);
+ inode-i_mtime.tv_sec = le32_to_cpu(sqsh_ino-mtime);
+ inode-i_atime.tv_sec = inode-i_mtime.tv_sec;
+ inode-i_ctime.tv_sec = inode-i_mtime.tv_sec;
+ inode-i_mode = le16_to_cpu(sqsh_ino-mode);
+ inode-i_size = 0;
+
+ return err;
+}
+
+
+struct inode *squashfs_iget(struct super_block *sb, long long ino,
+ unsigned int ino_number)
+{
+ struct inode *inode = iget_locked(sb, ino_number);
+ int err;
+
+ TRACE(Entered squashfs_iget\n);
+
+ if (!inode)
+ return ERR_PTR(-ENOMEM);
+ if (!(inode-i_state I_NEW))
+ return inode;
+
+ err = squashfs_read_inode(inode, ino);
+ if (err) {
+ iget_failed(inode);
+ return ERR_PTR(err);
+ }
+
+ unlock_new_inode(inode);
+ return inode;
+}
+
+
+/*
+ * Initialise VFS inode by reading inode from inode table (compressed
+ * metadata). The format and amount of data read depends on type.
+ */
+int squashfs_read_inode(struct inode *inode, long long ino)
+{
+ struct super_block *sb = inode-i_sb;
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ long long block = SQUASHFS_INODE_BLK(ino) + msblk-inode_table;
+ int err, type, offset = SQUASHFS_INODE_OFFSET(ino);
+ union squashfs_inode squashfs_ino;
+ struct squashfs_base_inode *sqshb_ino = squashfs_ino.base;
+
+ TRACE(Entered squashfs_read_inode\n);
+
+ /*
+* Read inode base common to all inode types.
+*/
+ err = squashfs_read_metadata(sb, sqshb_ino, block,
+ offset, sizeof(*sqshb_ino));
+ if (err 0)
+ goto failed_read;
+
+ err = squashfs_new_inode(sb, inode, sqshb_ino);
+ if (err)
+ goto failed_read;
+
+ block = SQUASHFS_INODE_BLK(ino) + msblk-inode_table;
+ offset = SQUASHFS_INODE_OFFSET(ino);
+
+ type = le16_to_cpu(sqshb_ino-inode_type);
+ switch (type) {
+ case SQUASHFS_REG_TYPE: {
+ unsigned int frag_offset, frag_size, frag;
+ long long frag_blk;
+ struct squashfs_reg_inode *sqsh_ino = squashfs_ino.reg
[PATCH V2 03/16] Squashfs: directory readdir operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/dir.c | 236 +
1 files changed, 236 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/dir.c b/fs/squashfs/dir.c
new file mode 100644
index 000..f5564e5
--- /dev/null
+++ b/fs/squashfs/dir.c
@@ -0,0 +1,236 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * dir.c
+ */
+
+/*
+ * This file implements code to read directories from disk.
+ *
+ * See namei.c for a description of directory organisation on disk.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+static const unsigned char squashfs_filetype_table[] = {
+ DT_UNKNOWN, DT_DIR, DT_REG, DT_LNK, DT_BLK, DT_CHR, DT_FIFO, DT_SOCK
+};
+
+/*
+ * Lookup offset (f_pos) in the directory index, returning the
+ * metadata block containing it.
+ *
+ * If we get an error reading the index then return the part of the index
+ * (if any) we have managed to read - the index isn't essential, just
+ * quicker.
+ */
+static int get_dir_index_using_offset(struct super_block *sb,
+ long long *next_block, int *next_offset,
+ long long index_start, int index_offset, int i_count,
+ long long f_pos)
+{
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ int err, i, index, length = 0;
+ struct squashfs_dir_index dir_index;
+
+ TRACE(Entered get_dir_index_using_offset, i_count %d, f_pos %lld\n,
+ i_count, f_pos);
+
+ /*
+* Translate from external f_pos to the internal f_pos. This
+* is offset by 3 because we invent . and .. entries which are
+* not actually stored in the directory.
+*/
+ if (f_pos 3)
+ return f_pos;
+ f_pos -= 3;
+
+ for (i = 0; i i_count; i++) {
+ err = squashfs_read_metadata(sb, dir_index, index_start,
+ index_offset, sizeof(dir_index));
+ if (err 0)
+ break;
+
+ index = le32_to_cpu(dir_index.index);
+ if (index f_pos)
+ /*
+* Found the index we're looking for.
+*/
+ break;
+
+ err = squashfs_read_metadata(sb, NULL, index_start,
+ index_offset, le32_to_cpu(dir_index.size) + 1);
+ if (err 0)
+ break;
+
+ length = index;
+ *next_block = le32_to_cpu(dir_index.start_block) +
+ msblk-directory_table;
+ }
+
+ *next_offset = (length + *next_offset) % SQUASHFS_METADATA_SIZE;
+
+ /*
+* Translate back from internal f_pos to external f_pos.
+*/
+ return length + 3;
+}
+
+
+static int squashfs_readdir(struct file *file, void *dirent, filldir_t filldir)
+{
+ struct inode *inode = file-f_dentry-d_inode;
+ struct squashfs_sb_info *msblk = inode-i_sb-s_fs_info;
+ long long block = SQUASHFS_I(inode)-start + msblk-directory_table;
+ int offset = SQUASHFS_I(inode)-offset, length = 0, dir_count, size,
+ type, err;
+ unsigned int inode_number;
+ struct squashfs_dir_header dirh;
+ struct squashfs_dir_entry *dire;
+
+ TRACE(Entered squashfs_readdir [%llx:%x]\n, block, offset);
+
+ dire = kmalloc(sizeof(*dire) + SQUASHFS_NAME_LEN + 1, GFP_KERNEL);
+ if (dire == NULL) {
+ ERROR(Failed to allocate squashfs_dir_entry\n);
+ goto finish;
+ }
+
+ /*
+* Return . and .. entries as the first two filenames in the
+* directory. To maximise compression these two entries are not
+* stored in the directory, and so we invent them here.
+*
+* It also means that the external f_pos is offset by 3 from the
+* on-disk directory f_pos.
+*/
+ while (file-f_pos 3
[PATCH V2 13/16] Squashfs: Makefiles
Signed-off-by: Phillip Lougher [EMAIL PROTECTED] --- fs/Makefile |1 + fs/squashfs/Makefile |8 2 files changed, 9 insertions(+), 0 deletions(-) diff --git a/fs/Makefile b/fs/Makefile index 2168c90..90c1eee 100644 --- a/fs/Makefile +++ b/fs/Makefile @@ -76,6 +76,7 @@ obj-$(CONFIG_JBD) += jbd/ obj-$(CONFIG_JBD2) += jbd2/ obj-$(CONFIG_EXT2_FS) += ext2/ obj-$(CONFIG_CRAMFS) += cramfs/ +obj-$(CONFIG_SQUASHFS) += squashfs/ obj-y += ramfs/ obj-$(CONFIG_HUGETLBFS)+= hugetlbfs/ obj-$(CONFIG_CODA_FS) += coda/ diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile new file mode 100644 index 000..8258cf9 --- /dev/null +++ b/fs/squashfs/Makefile @@ -0,0 +1,8 @@ +# +# Makefile for the linux squashfs routines. +# + +obj-$(CONFIG_SQUASHFS) += squashfs.o +squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o +squashfs-y += namei.o super.o symlink.o +#squashfs-y += squashfs2_0.o -- 1.5.2.5 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V2 05/16] Squashfs: symlink operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/symlink.c | 119 +
1 files changed, 119 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/symlink.c b/fs/squashfs/symlink.c
new file mode 100644
index 000..b031a4e
--- /dev/null
+++ b/fs/squashfs/symlink.c
@@ -0,0 +1,119 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * symlink.c
+ */
+
+/*
+ * This file implements code to handle symbolic links.
+ *
+ * The data contents of symbolic links are stored inside the symbolic
+ * link inode within the inode table. This allows the normally small symbolic
+ * link to be compressed as part of the inode table, achieving much greater
+ * compression than if the symbolic link was compressed individually.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/kernel.h
+#include linux/slab.h
+#include linux/string.h
+#include linux/pagemap.h
+#include linux/zlib.h
+
+#include squashfs_fs.h
+#include squashfs_fs_sb.h
+#include squashfs_fs_i.h
+#include squashfs.h
+
+static int squashfs_symlink_readpage(struct file *file, struct page *page)
+{
+ struct inode *inode = page-mapping-host;
+ struct super_block *sb = inode-i_sb;
+ struct squashfs_sb_info *msblk = sb-s_fs_info;
+ int index = page-index PAGE_CACHE_SHIFT;
+ long long block = SQUASHFS_I(inode)-start;
+ int offset = SQUASHFS_I(inode)-offset;
+ int length = min_t(int, i_size_read(inode) - index, PAGE_CACHE_SIZE);
+ int avail, bytes;
+ void *pageaddr;
+ struct squashfs_cache_entry *entry;
+
+ TRACE(Entered squashfs_symlink_readpage, page index %ld, start block
+ %llx, offset %x\n, page-index, block, offset);
+
+ /*
+* Skip index bytes into symlink metadata.
+*/
+ if (index) {
+ bytes = squashfs_read_metadata(sb, NULL, block, offset,
+ index);
+ if (bytes 0) {
+ ERROR(Unable to read symlink [%llx:%x]\n,
+ SQUASHFS_I(inode)-start,
+ SQUASHFS_I(inode)-offset);
+ goto error_out;
+ }
+ }
+
+ /*
+* Read length bytes from symlink metadata. Squashfs_read_metadata
+* is not used here because it can sleep and we want to use
+* kmap_atomic to map the page. Instead call the underlying
+* squashfs_cache_get routine. As length bytes may overlap metadata
+* blocks, we may need to call squashfs_cache_get multiple times.
+*/
+ for (bytes = 0; bytes length; offset = 0, bytes += avail) {
+ entry = squashfs_cache_get(sb, msblk-block_cache, block, 0);
+ avail = min(entry-length - offset, length - bytes);
+
+ if (entry-error) {
+ ERROR(Unable to read symlink [%llx:%x]\n,
+ SQUASHFS_I(inode)-start,
+ SQUASHFS_I(inode)-offset);
+ squashfs_cache_put(msblk-block_cache, entry);
+ goto error_out;
+ }
+
+ pageaddr = kmap_atomic(page, KM_USER0);
+ memcpy(pageaddr + bytes, entry-data + offset, avail);
+ if (avail == length - bytes)
+ memset(pageaddr + length, 0, PAGE_CACHE_SIZE - length);
+ else
+ block = entry-next_index;
+ kunmap_atomic(pageaddr, KM_USER0);
+ squashfs_cache_put(msblk-block_cache, entry);
+ }
+
+ flush_dcache_page(page);
+ SetPageUptodate(page);
+ unlock_page(page);
+ return 0;
+
+error_out:
+ SetPageError(page);
+ unlock_page(page);
+ return 0;
+}
+
+
+const struct address_space_operations squashfs_symlink_aops = {
+ .readpage = squashfs_symlink_readpage
+};
--
1.5.2.5
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http
Re: Subject: [PATCH 00/16] Squashfs: compressed read-only filesystem
Geert Uytterhoeven wrote: Hi Phillip, On Fri, 17 Oct 2008, Phillip Lougher wrote: This is a second attempt at mainlining Squashfs. The first attempt was way This is great news! I ran a quick test of squashfs 4.0 (the CVS version) on UML/ia32 and ppc64, and it seems to work fine! Great job! Let's hope we'll see it in mainline soon... Thanks! I hope it gets into mainline soon too :) BTW, one minor gripe is that the current mksquashfs doesn't want to run on big endian yet, as there's no byteswapping support. Yeah, I know about that. There's still some work needing to be done on the squashfs-tools. I figured it was important to get the kernel stuff submitted and discussed ASAP. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Subject: [PATCH 01/16] Squashfs: inode operations
Geert Uytterhoeven wrote: Sparse with endian checking (make C=2 CHECKFLAGS=-D__CHECK_ENDIAN__) complains aibout these: | fs/squashfs/inode.c:306:25: warning: cast to restricted __le16 | fs/squashfs/inode.c:324:25: warning: cast to restricted __le16 and it seems to be right, as inode.i_mode is not __le16. I think the le16_to_cpu() should be removed. Yes, you're right. Fixed thanks. BTW, there are also a few sparse warnings about different signednesses, so you probably want to run sparse yourself, too. I'll do that. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Subject: [PATCH 01/16] Squashfs: inode operations
Jörn Engel wrote: None of the comments below are a reason against mainline inclusion, imo. They should get handled, but whether that happens before or after a merge doesn't really matter. Yeah you're right regarding your comments. That's where code-review comes in handy, to spot things you don't see because you're too used to the code. I'm working on a re-spin incorporating your comments, and it should be ready tomorrow. Thanks for the code-review. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Subject: [PATCH 00/16] Squashfs: compressed read-only filesystem
David P. Quigley wrote: Looking through the code I see two references to xattrs, one is the index of the xattr table in the superblock and there seems to be struct member in one of the inode structures that is an index into this table. Looking through the code I don't see either of these used at all. Do you intend to add xattr support at some point? I saw reference to the desire to add xattr support in an email from 2004 but you said that the code has been rewritten since then. If you are going to add xattr support you probably want to add it to more than just regular files. In SELinux and other LSMs symlinks and directories are also labeled so they will need xattr entries. Yes and yes. I am intending to add xattr support, something that's been on my to-do list for a long time (since 2004 as you said), but it's been something which I've never got the time to do. Once (if) Squashfs is mainlined, it will be the next thing. The xattr references in the layout is my attempt at forward planning to avoid making an incompatible layout change when I finally get around to implementing it. My plan is to put xattrs in a table (referenced by the superblock), and then put indexes in extended inodes which index into the table (as you noticed). The general idea in Squashfs is that inodes get optimised for normally occurring cases, and less common cases (that would need a bigger inode) get to use an extended inode. Squashfs currently has an extended regular file inode, which is where the xattr index will sit, and so this has had an xattr index added. The other inodes don't currently have extended inodes, these will be defined when I implement xattrs (which is why they're missing). Having said that, I've fscked up and forgotten to add an xattr field to the extended directory inode which is currently defined :) Thanks for spotting this. Phillip Dave -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Subject: [PATCH 00/16] Squashfs: compressed read-only filesystem
David P. Quigley wrote: In SELinux and other LSMs symlinks and directories are also labeled so they will need xattr entries. BTW you don't mention device, fifo and socket inodes... Do they ever get labelled? It's something I was going to look into closer to an implementation, but it would be interesting to know. Phillip Dave -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Subject: [PATCH 14/16] Squashfs: Kconfig entry
Signed-off-by: Phillip Lougher [EMAIL PROTECTED] --- fs/Kconfig | 52 1 files changed, 52 insertions(+), 0 deletions(-) diff --git a/fs/Kconfig b/fs/Kconfig index cf9db09..abbb6c2 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -1192,6 +1192,58 @@ config CRAMFS If unsure, say N. +config SQUASHFS + tristate SquashFS 4.0 - Squashed file system support + depends on BLOCK + select ZLIB_INFLATE + help + Saying Y here includes support for SquashFS 4.0 (a Compressed + Read-Only File System). Squashfs is a highly compressed read-only + filesystem for Linux. It uses zlib compression to compress both + files, inodes and directories. Inodes in the system are very small + and all blocks are packed to minimise data overhead. Block sizes + greater than 4K are supported up to a maximum of 1 Mbytes (default + block size 128K). SquashFS 4.0 supports 64 bit filesystems and files + (larger than 4GB), full uid/gid information, hard links and + timestamps. + + Squashfs is intended for general read-only filesystem use, for + archival use (i.e. in cases where a .tar.gz file may be used), and in + embedded systems where low overhead is needed. Further information + and tools are available from http://squashfs.sourceforge.net. + + If you want to compile this as a module ( = code which can be + inserted in and removed from the running kernel whenever you want), + say M here and read file:Documentation/modules.txt. The module + will be called squashfs. Note that the root file system (the one + containing the directory /) cannot be compiled as a module. + + If unsure, say N. + +config SQUASHFS_EMBEDDED + + bool Additional option for memory-constrained systems + depends on SQUASHFS + default n + help + Saying Y here allows you to specify cache size. + + If unsure, say N. + +config SQUASHFS_FRAGMENT_CACHE_SIZE + int Number of fragments cached if SQUASHFS_EMBEDDED + depends on SQUASHFS + default 3 + help + By default SquashFS caches the last 3 fragments read from + the filesystem. Increasing this amount may mean SquashFS + has to re-read fragments less often from disk, at the expense + of extra system memory. Decreasing this amount will mean + SquashFS uses less memory at the expense of extra reads from disk. + + Note there must be at least one cached fragment. Anything + much more than three will probably not make much difference. + config VXFS_FS tristate FreeVxFS file system support (VERITAS VxFS(TM) compatible) depends on BLOCK -- 1.5.2.5 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Subject: [PATCH 12/16] Squashfs: header files
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/squashfs.h | 100 +++
include/linux/squashfs_fs.h| 383
include/linux/squashfs_fs_i.h | 45 +
include/linux/squashfs_fs_sb.h | 76
4 files changed, 604 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/squashfs.h b/fs/squashfs/squashfs.h
new file mode 100644
index 000..711cd43
--- /dev/null
+++ b/fs/squashfs/squashfs.h
@@ -0,0 +1,100 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * squashfs.h
+ */
+
+#ifdef SQUASHFS_TRACE
+#define TRACE(s, args...) printk(KERN_NOTICE SQUASHFS: s, ## args)
+#else
+#define TRACE(s, args...) {}
+#endif
+
+#define ERROR(s, args...) printk(KERN_ERR SQUASHFS error: s, ## args)
+
+#define SERROR(s, args...) \
+ do { \
+ if (!silent) \
+ printk(KERN_ERR SQUASHFS error: s, ## args);\
+ } while (0)
+
+#define WARNING(s, args...)printk(KERN_WARNING SQUASHFS: s, ## args)
+
+static inline struct squashfs_inode_info *SQUASHFS_I(struct inode *inode)
+{
+ return list_entry(inode, struct squashfs_inode_info, vfs_inode);
+}
+
+/* block.c */
+extern unsigned int squashfs_read_data(struct super_block *, void *,
+ long long, unsigned int, long long *, int);
+
+/* cache.c */
+extern struct squashfs_cache *squashfs_cache_init(char *, int, int, int);
+extern void squashfs_cache_delete(struct squashfs_cache *);
+struct squashfs_cache_entry *squashfs_cache_get(struct super_block *,
+ struct squashfs_cache *, long long, int);
+void squashfs_cache_put(struct squashfs_cache *, struct squashfs_cache_entry
*);
+extern int squashfs_read_metadata(struct super_block *, void *,
+ long long, unsigned int, int, long long *,
+ unsigned int *);
+extern struct squashfs_cache_entry *get_cached_fragment(struct super_block *,
+ long long, int);
+extern void release_cached_fragment(struct squashfs_sb_info *,
+ struct squashfs_cache_entry *);
+
+/* export.c */
+extern __le64 *read_inode_lookup_table(struct super_block *, long long,
+ unsigned int);
+
+/* fragment.c */
+extern int get_fragment_location(struct super_block *, unsigned int,
+ long long *);
+extern __le64 *read_fragment_index_table(struct super_block *, long long,
+ unsigned int);
+
+/* id.c */
+extern int squashfs_get_id(struct super_block *, unsigned int, unsigned int *);
+extern __le64 *read_id_index_table(struct super_block *, long long,
+ unsigned short);
+
+/* inode.c */
+extern struct inode *squashfs_iget(struct super_block *, long long,
+ unsigned int);
+extern int squashfs_read_inode(struct inode *, long long);
+
+/*
+ * Inodes and files operations
+ */
+
+/* dir.c */
+extern const struct file_operations squashfs_dir_ops;
+
+/* export.c */
+extern const struct export_operations squashfs_export_ops;
+
+/* file.c */
+extern const struct address_space_operations squashfs_aops;
+
+/* namei.c */
+extern const struct inode_operations squashfs_dir_inode_ops;
+
+/* symlink.c */
+extern const struct address_space_operations squashfs_symlink_aops;
diff --git a/include/linux/squashfs_fs.h b/include/linux/squashfs_fs.h
new file mode 100644
index 000..aeb902f
--- /dev/null
+++ b/include/linux/squashfs_fs.h
@@ -0,0 +1,383 @@
+#ifndef SQUASHFS_FS
+#define SQUASHFS_FS
+/*
+ * Squashfs
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty
Subject: [PATCH 08/16] Squashfs: fragment block operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/fragment.c | 89
1 files changed, 89 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/fragment.c b/fs/squashfs/fragment.c
new file mode 100644
index 000..4a23e83
--- /dev/null
+++ b/fs/squashfs/fragment.c
@@ -0,0 +1,89 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * fragment.c
+ */
+
+/*
+ * This file implements code to handle compressed fragments (tail-end packed
+ * datablocks).
+ *
+ * Regular files contain a fragment index which is mapped to a fragment
+ * location on disk and compressed size using a fragment lookup table.
+ * Like everything in Squashfs this fragment lookup table is itself stored
+ * compressed into metadata blocks. A second index table is used to locate
+ * these. This second index table for speed of access (and because it
+ * is small) is read at mount time and cached in memory.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/zlib.h
+#include linux/squashfs_fs.h
+#include linux/squashfs_fs_sb.h
+#include linux/squashfs_fs_i.h
+
+#include squashfs.h
+
+int get_fragment_location(struct super_block *s, unsigned int fragment,
+ long long *fragment_block)
+{
+ struct squashfs_sb_info *msblk = s-s_fs_info;
+ int block = SQUASHFS_FRAGMENT_INDEX(fragment);
+ int offset = SQUASHFS_FRAGMENT_INDEX_OFFSET(fragment);
+ long long start_block = le64_to_cpu(msblk-fragment_index[block]);
+ struct squashfs_fragment_entry fragment_entry;
+ int size = 0;
+
+ if (!squashfs_read_metadata(s, fragment_entry, start_block, offset,
+sizeof(fragment_entry), start_block, offset))
+ goto out;
+
+ *fragment_block = le64_to_cpu(fragment_entry.start_block);
+ size = le32_to_cpu(fragment_entry.size);
+
+out:
+ return size;
+}
+
+
+__le64 *read_fragment_index_table(struct super_block *s,
+ long long fragment_table_start, unsigned int fragments)
+{
+ unsigned int length = SQUASHFS_FRAGMENT_INDEX_BYTES(fragments);
+ __le64 *fragment_index;
+
+ /* Allocate fragment index table */
+ fragment_index = kmalloc(length, GFP_KERNEL);
+ if (fragment_index == NULL) {
+ ERROR(Failed to allocate fragment index table\n);
+ return NULL;
+ }
+
+ if (!squashfs_read_data(s, fragment_index, fragment_table_start,
+ length | SQUASHFS_COMPRESSED_BIT_BLOCK, NULL, length)) {
+ ERROR(unable to read fragment index table\n);
+ kfree(fragment_index);
+ return NULL;
+ }
+
+ return fragment_index;
+}
--
1.5.2.5
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Subject: [PATCH 15/16] Squashfs: initrd support
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
init/do_mounts_rd.c | 14 ++
1 files changed, 14 insertions(+), 0 deletions(-)
diff --git a/init/do_mounts_rd.c b/init/do_mounts_rd.c
index fedef93..30086a7 100644
--- a/init/do_mounts_rd.c
+++ b/init/do_mounts_rd.c
@@ -5,6 +5,7 @@
#include linux/ext2_fs.h
#include linux/romfs_fs.h
#include linux/cramfs_fs.h
+#include linux/squashfs_fs.h
#include linux/initrd.h
#include linux/string.h
@@ -41,6 +42,7 @@ static int __init crd_load(int in_fd, int out_fd);
* ext2
* romfs
* cramfs
+ * squashfs
* gzip
*/
static int __init
@@ -51,6 +53,7 @@ identify_ramdisk_image(int fd, int start_block)
struct ext2_super_block *ext2sb;
struct romfs_super_block *romfsb;
struct cramfs_super *cramfsb;
+ struct squashfs_super_block *squashfsb;
int nblocks = -1;
unsigned char *buf;
@@ -62,6 +65,7 @@ identify_ramdisk_image(int fd, int start_block)
ext2sb = (struct ext2_super_block *) buf;
romfsb = (struct romfs_super_block *) buf;
cramfsb = (struct cramfs_super *) buf;
+ squashfsb = (struct squashfs_super_block *) buf;
memset(buf, 0xe5, size);
/*
@@ -99,6 +103,16 @@ identify_ramdisk_image(int fd, int start_block)
goto done;
}
+ /* squashfs is at block zero too */
+ if (le32_to_cpu(squashfsb-s_magic) == SQUASHFS_MAGIC) {
+ printk(KERN_NOTICE
+ RAMDISK: squashfs filesystem found at block %d\n,
+ start_block);
+ nblocks = (le64_to_cpu(squashfsb-bytes_used) + BLOCK_SIZE - 1)
+ BLOCK_SIZE_BITS;
+ goto done;
+ }
+
/*
* Read block 1 to test for minix and ext2 superblock
*/
--
1.5.2.5
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Subject: [PATCH 11/16] Squashfs: block operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/block.c | 257 +++
1 files changed, 257 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
new file mode 100644
index 000..0d1a74d
--- /dev/null
+++ b/fs/squashfs/block.c
@@ -0,0 +1,257 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * block.c
+ */
+
+/*
+ * This file implements the low-level routines to read and decompress
+ * datablocks and metadata blocks.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/mutex.h
+#include linux/string.h
+#include linux/buffer_head.h
+#include linux/zlib.h
+#include linux/squashfs_fs.h
+#include linux/squashfs_fs_sb.h
+#include linux/squashfs_fs_i.h
+
+#include squashfs.h
+
+/*
+ * Read the metadata block length, this is stored in the first two
+ * bytes of the metadata block.
+ */
+static struct buffer_head *get_block_length(struct super_block *s,
+ int *cur_index, int *offset, unsigned int *length)
+{
+ struct squashfs_sb_info *msblk = s-s_fs_info;
+ struct buffer_head *bh;
+
+ bh = sb_bread(s, *cur_index);
+ if (bh == NULL)
+ goto out;
+
+ if (msblk-devblksize - *offset == 1) {
+ *length = (unsigned char) bh-b_data[*offset];
+ brelse(bh);
+ bh = sb_bread(s, ++(*cur_index));
+ if (bh == NULL)
+ goto out;
+ *length |= (unsigned char) bh-b_data[0] 8;
+ *offset = 1;
+ } else {
+ *length = (unsigned char) bh-b_data[*offset] |
+ (unsigned char) bh-b_data[*offset + 1] 8;
+ *offset += 2;
+ }
+
+out:
+ return bh;
+}
+
+
+/*
+ * Read and decompress a metadata block or datablock. Length is non-zero
+ * if a datablock is being read (the size is stored elsewhere in the
+ * filesystem), otherwise the length is obtained from the first two bytes of
+ * the metadata block. A bit in the length field indicates if the block
+ * is stored uncompressed in the filesystem (usually because compression
+ * generated a larger block - this does occasionally happen with zlib).
+ */
+unsigned int squashfs_read_data(struct super_block *s, void *buffer,
+ long long index, unsigned int length,
+ long long *next_index, int srclength)
+{
+ struct squashfs_sb_info *msblk = s-s_fs_info;
+ struct buffer_head **bh;
+ unsigned int offset = index ((1 msblk-devblksize_log2) - 1);
+ unsigned int cur_index = index msblk-devblksize_log2;
+ int bytes, avail, b = 0, k = 0;
+ unsigned int compressed;
+ unsigned int c_byte = length;
+
+ bh = kcalloc((msblk-block_size msblk-devblksize_log2) + 1,
+ sizeof(*bh), GFP_KERNEL);
+ if (bh == NULL)
+ goto read_failure;
+
+ if (c_byte) {
+ /*
+* Datablock.
+*/
+ bytes = -offset;
+ compressed = SQUASHFS_COMPRESSED_BLOCK(c_byte);
+ c_byte = SQUASHFS_COMPRESSED_SIZE_BLOCK(c_byte);
+
+ TRACE(Block @ 0x%llx, %scompressed size %d, src size %d\n,
+ index, compressed ? : un, c_byte, srclength);
+
+ if (c_byte srclength || index 0 ||
+ (index + c_byte) msblk-bytes_used)
+ goto read_failure;
+
+ for (b = 0; bytes (int) c_byte; b++, cur_index++) {
+ bh[b] = sb_getblk(s, cur_index);
+ if (bh[b] == NULL)
+ goto block_release;
+ bytes += msblk-devblksize;
+ }
+ ll_rw_block(READ, b, bh);
+ } else {
+ /*
+* Metadata block.
+*/
+ if (index 0 || (index + 2) msblk-bytes_used)
+ goto read_failure;
+
+ bh[0] = get_block_length(s, cur_index, offset, c_byte
Subject: [PATCH 13/14] Squashfs: Makefiles
Signed-off-by: Phillip Lougher [EMAIL PROTECTED] --- fs/Makefile |1 + fs/squashfs/Makefile |8 2 files changed, 9 insertions(+), 0 deletions(-) diff --git a/fs/Makefile b/fs/Makefile index b6f27dc..42d881b 100644 --- a/fs/Makefile +++ b/fs/Makefile @@ -75,6 +75,7 @@ obj-$(CONFIG_JBD) += jbd/ obj-$(CONFIG_JBD2) += jbd2/ obj-$(CONFIG_EXT2_FS) += ext2/ obj-$(CONFIG_CRAMFS) += cramfs/ +obj-$(CONFIG_SQUASHFS) += squashfs/ obj-y += ramfs/ obj-$(CONFIG_HUGETLBFS)+= hugetlbfs/ obj-$(CONFIG_CODA_FS) += coda/ diff --git a/fs/squashfs/Makefile b/fs/squashfs/Makefile new file mode 100644 index 000..8258cf9 --- /dev/null +++ b/fs/squashfs/Makefile @@ -0,0 +1,8 @@ +# +# Makefile for the linux squashfs routines. +# + +obj-$(CONFIG_SQUASHFS) += squashfs.o +squashfs-y += block.o cache.o dir.o export.o file.o fragment.o id.o inode.o +squashfs-y += namei.o super.o symlink.o +#squashfs-y += squashfs2_0.o -- 1.5.2.5 -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Subject: [PATCH 09/16] Squashfs: uid/gid lookup operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/id.c | 84 ++
1 files changed, 84 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/id.c b/fs/squashfs/id.c
new file mode 100644
index 000..b4881d7
--- /dev/null
+++ b/fs/squashfs/id.c
@@ -0,0 +1,84 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * id.c
+ */
+
+/*
+ * This file implements code to handle uids and gids.
+ *
+ * For space efficiency inodes store uid and gid indexes, which are
+ * converted to 32-bit uids/gids using an id look up table. This table is
+ * stored compressed into metadata blocks. A second index table is used to
+ * locate these. This second index table for speed of access (and because it
+ * is small) is read at mount time and cached in memory.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/zlib.h
+#include linux/squashfs_fs.h
+#include linux/squashfs_fs_sb.h
+#include linux/squashfs_fs_i.h
+
+#include squashfs.h
+
+int squashfs_get_id(struct super_block *s, unsigned int index, unsigned int
*id)
+{
+ struct squashfs_sb_info *msblk = s-s_fs_info;
+ int block = SQUASHFS_ID_BLOCK(index);
+ int offset = SQUASHFS_ID_BLOCK_OFFSET(index);
+ long long start_block = le64_to_cpu(msblk-id_table[block]);
+ __le32 disk_id;
+
+ if (!squashfs_read_metadata(s, disk_id, start_block, offset,
+sizeof(disk_id), start_block, offset))
+ return 0;
+
+ *id = le32_to_cpu(disk_id);
+ return 1;
+}
+
+
+__le64 *read_id_index_table(struct super_block *s, long long id_table_start,
+ unsigned short no_ids)
+{
+ unsigned int length = SQUASHFS_ID_BLOCK_BYTES(no_ids);
+ __le64 *id_table;
+
+ TRACE(In read_id_index_table, length %d\n, length);
+
+ /* Allocate id index table */
+ id_table = kmalloc(length, GFP_KERNEL);
+ if (id_table == NULL) {
+ ERROR(Failed to allocate id index table\n);
+ return NULL;
+ }
+
+ if (!squashfs_read_data(s, id_table, id_table_start, length |
+ SQUASHFS_COMPRESSED_BIT_BLOCK, NULL, length)) {
+ ERROR(unable to read id index table\n);
+ kfree(id_table);
+ return NULL;
+ }
+
+ return id_table;
+}
--
1.5.2.5
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Subject: [PATCH 02/16] Squashfs: directory lookup operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/namei.c | 226 +++
1 files changed, 226 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/namei.c b/fs/squashfs/namei.c
new file mode 100644
index 000..fdc1b03
--- /dev/null
+++ b/fs/squashfs/namei.c
@@ -0,0 +1,226 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * namei.c
+ */
+
+/*
+ * This file implements code to do filename lookup in directories.
+ *
+ * Like inodes, directories are packed into compressed metadata blocks, stored
+ * in a directory table. Directories are accessed using the start address of
+ * the metablock containing the directory and the offset into the
+ * decompressed block (block, offset).
+ *
+ * Directories are organised in a slightly complex way, and are not simply
+ * a list of file names. The organisation takes advantage of the
+ * fact that (in most cases) the inodes of the files will be in the same
+ * compressed metadata block, and therefore, can share the start block.
+ * Directories are therefore organised in a two level list, a directory
+ * header containing the shared start block value, and a sequence of directory
+ * entries, each of which share the shared start block. A new directory header
+ * is written once/if the inode start block changes. The directory
+ * header/directory entry list is repeated as many times as necessary.
+ *
+ * Directories are sorted, and can contain a directory index to speed up
+ * file lookup. Directory indexes store one entry per metablock, each entry
+ * storing the index/filename mapping to the first directory header
+ * in each metadata block. Directories are sorted in alphabetical order,
+ * and at lookup the index is scanned linearly looking for the first filename
+ * alphabetically larger than the filename being looked up. At this point the
+ * location of the metadata block the filename is in has been found.
+ * The general idea of the index is ensure only one metadata block needs to be
+ * decompressed to do a lookup irrespective of the length of the directory.
+ * This scheme has the advantage that it doesn't require extra memory overhead
+ * and doesn't require much extra storage on disk.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/string.h
+#include linux/dcache.h
+#include linux/zlib.h
+#include linux/squashfs_fs.h
+#include linux/squashfs_fs_sb.h
+#include linux/squashfs_fs_i.h
+
+#include squashfs.h
+
+/*
+ * Lookup name in the directory index, returning the location of the metadata
+ * block containing it, and the directory index this represents.
+ */
+static int get_dir_index_using_name(struct super_block *s,
+ long long *next_block, unsigned int *next_offset,
+ long long index_start, unsigned int index_offset,
+ int i_count, const char *name, int len)
+{
+ struct squashfs_sb_info *msblk = s-s_fs_info;
+ int i, size, length = 0;
+ struct squashfs_dir_index *index;
+ char *str;
+
+ TRACE(Entered get_dir_index_using_name, i_count %d\n, i_count);
+
+ str = kmalloc(sizeof(*index) + (SQUASHFS_NAME_LEN + 1) * 2, GFP_KERNEL);
+ if (str == NULL) {
+ ERROR(Failed to allocate squashfs_dir_index\n);
+ goto out;
+ }
+
+ index = (struct squashfs_dir_index *) (str + SQUASHFS_NAME_LEN + 1);
+ strncpy(str, name, len);
+ str[len] = '\0';
+
+ for (i = 0; i i_count; i++) {
+ squashfs_read_metadata(s, index, index_start, index_offset,
+ sizeof(*index), index_start,
+ index_offset);
+
+ size = le32_to_cpu(index-size) + 1;
+
+ squashfs_read_metadata(s, index-name, index_start,
+ index_offset, size, index_start,
+ index_offset);
+
+ index-name[size] = '\0';
+
+ if (strcmp(index-name, str) 0)
+ break;
+
+ length = le32_to_cpu(index
Subject: [PATCH 00/16] Squashfs: compressed read-only filesystem
This is a second attempt at mainlining Squashfs. The first attempt was way way back in early 2005 :-) Since then the filesystem layout has undergone two major revisions, and the kernel code has almost been completely rewritten. Both of these were to address the criticisms made at the original attempt. Summary of changes: 1. Filesystem layout is now 64-bit, in theory filesystems and files can be 2^64 in size. 2. Filesystem is now fixed little-endian. 3. . and .. are now returned by readdir. 4. Sparse files are now supported. 5. Filesystem is now exportable (NFS etc.). 6. Datablocks up to 1 Mbyte are now supported. Codewise all of the packed bit-fields and the swap macros have been removed in favour of aligned structures and in-line swapping using leXX_to_cpu(). The code has also been extensively restructured, reformatted to kernel coding standards and commented. Previously there was resistance to the inclusion of another compressed filesystem when Linux already had cramfs. There was pressure for a strong case to be made for the inclusion of Squashfs. Hopefully the case for the inclusion of other compressed filesystems has now already been answered over the last couple of years, however, it is worth listing the features of Squashfs over cramfs, which is still the only read-only compressed filesystem in mainline. Max filesystem size: cramfs 16 Mbytes, Squashfs 64-bit filesystem Max filesize: cramfs 16 Mbytes, Squashfs 64-bit filesystem Block size: cramfs 4K, Squashfs default 128K, max 1Mbyte Tail-end packing: cramfs no, Squashfs yes Directory indexes: cramfs no, Squashfs yes Compressed metadata: cramfs no, Squashfs yes Hard link support: cramfs no, Squashfs yes Support for . and .. in readdir: cramfs no, Squashfs yes Real inode numbers: cramfs no, Squashfs yes. Cramfs gives device inodes, fifo and empty directories the same inode of 1! Exportable filesystem (NFS, etc.): cramfs no, Squashfs yes Active maintenance: cramfs no (it is listed as orphaned, probably no active work for years), Squashfs yes Sorry for the list formatting, but many email readers are very unforgiving displaying tabbed lists and so I avoided them. For those that want hard performance statistics http://tree.celinuxforum.org/CelfPubWiki/SquashFsComparisons gives a full comparison of the performance of Squashfs against cramfs, zisofs, cloop and ext3. I made these tests a number of years ago using Squashfs 2.1, but they are still valid. In fact the performance should now be better. Cramfs is a limited filesystem, it's good for some embedded users but not now much else, its layout and features hasn't changed in the eight years+ since its release. Squashfs, despite never being in mainline, has been actively developed for over six years, and in that time has gone through four layout revisions, each revision improving compression and performance where limitations were found. For an often dismissed filesystem, Squashfs has advanced features such as metadata compression and tail-end packing for greater compression, and directory indexes for faster dentry operations. Despite not being in mainline, it is widely used. It is packaged by all major distributions (Ubuntu, Fedora, Debian, SUSE, Gentoo), it is used on most LiveCDs, it is extensively used in embedded systems (STBs, routers, mobile phones), and notably is used in such things as Splashtop and the Amazon Kindle. Anyway that's my case for inclusion. If any readers want Squashfs mainlined it's probably now a good time to offer support! There are 16 patches in the patch set, and the patches are against the latest linux-next tree (linux 2.6.27-next-20081016). Finally, I would like to acknowledge the financial support of the Consumer Embedded Linux Forum (CELF). They've made it possible for me to spend the last four months working full time on this mainlining attempt. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Subject: [PATCH 10/16] Squashfs: cache operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/cache.c | 316 +++
1 files changed, 316 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/cache.c b/fs/squashfs/cache.c
new file mode 100644
index 000..0fe994f
--- /dev/null
+++ b/fs/squashfs/cache.c
@@ -0,0 +1,316 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * cache.c
+ */
+
+/*
+ * Blocks in Squashfs are compressed. To avoid repeatedly decompressing
+ * recently accessed data Squashfs uses two small metadata and fragment caches.
+ *
+ * This file implements a generic cache implementation used for both caches,
+ * plus functions layered ontop of the generic cache implementation to
+ * access the metadata and fragment caches.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/vmalloc.h
+#include linux/sched.h
+#include linux/spinlock.h
+#include linux/wait.h
+#include linux/zlib.h
+#include linux/squashfs_fs.h
+#include linux/squashfs_fs_sb.h
+#include linux/squashfs_fs_i.h
+
+#include squashfs.h
+
+/*
+ * Look-up block in cache, and increment usage count. If not in cache, read
+ * and decompress it from disk.
+ */
+struct squashfs_cache_entry *squashfs_cache_get(struct super_block *s,
+ struct squashfs_cache *cache, long long block, int length)
+{
+ int i, n;
+ struct squashfs_cache_entry *entry;
+
+ spin_lock(cache-lock);
+
+ while (1) {
+ for (i = 0; i cache-entries; i++)
+ if (cache-entry[i].block == block)
+ break;
+
+ if (i == cache-entries) {
+ /*
+* Block not in cache, if all cache entries are locked
+* go to sleep waiting for one to become available.
+*/
+ if (cache-unused == 0) {
+ cache-waiting++;
+ spin_unlock(cache-lock);
+ wait_event(cache-wait_queue, cache-unused);
+ spin_lock(cache-lock);
+ cache-waiting--;
+ continue;
+ }
+
+ /*
+* At least one unlocked cache entry. A simple
+* round-robin strategy is used to choose the entry to
+* be evicted from the cache.
+*/
+ i = cache-next_blk;
+ for (n = 0; n cache-entries; n++) {
+ if (cache-entry[i].locked == 0)
+ break;
+ i = (i + 1) % cache-entries;
+ }
+
+ cache-next_blk = (i + 1) % cache-entries;
+ entry = cache-entry[i];
+
+ /*
+* Initialise choosen cache entry, and fill it in from
+* disk.
+*/
+ cache-unused--;
+ entry-block = block;
+ entry-locked = 1;
+ entry-pending = 1;
+ entry-waiting = 0;
+ entry-error = 0;
+ spin_unlock(cache-lock);
+
+ entry-length = squashfs_read_data(s, entry-data,
+ block, length, entry-next_index,
+ cache-block_size);
+
+ spin_lock(cache-lock);
+
+ if (entry-length == 0)
+ entry-error = 1;
+
+ entry-pending = 0;
+ spin_unlock(cache-lock);
+
+ /*
+* While filling this entry one or more other processes
+* have looked it up in the cache, and have slept
+* waiting for it to become available.
+*/
+ if (entry
Subject: [PATCH 06/16] Squashfs: super block operations
Signed-off-by: Phillip Lougher [EMAIL PROTECTED]
---
fs/squashfs/super.c | 411 +++
1 files changed, 411 insertions(+), 0 deletions(-)
diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c
new file mode 100644
index 000..a374271
--- /dev/null
+++ b/fs/squashfs/super.c
@@ -0,0 +1,411 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
+ * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008
+ * Phillip Lougher [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * super.c
+ */
+
+/*
+ * This file implements code to read the superblock, read and initialise
+ * in-memory structures at mount time, and all the VFS glue code to register
+ * the filesystem.
+ */
+
+#include linux/fs.h
+#include linux/vfs.h
+#include linux/slab.h
+#include linux/vmalloc.h
+#include linux/mutex.h
+#include linux/pagemap.h
+#include linux/init.h
+#include linux/module.h
+#include linux/zlib.h
+#include linux/squashfs_fs.h
+#include linux/squashfs_fs_sb.h
+#include linux/squashfs_fs_i.h
+
+#include squashfs.h
+
+static struct file_system_type squashfs_fs_type;
+static struct super_operations squashfs_super_ops;
+
+static int supported_squashfs_filesystem(short major, short minor,
+ short compression)
+{
+ if (major SQUASHFS_MAJOR) {
+ ERROR(Major/Minor mismatch, older Squashfs %d.%d filesystems
+ are unsupported\n, major, minor);
+ return 0;
+ } else if (major SQUASHFS_MAJOR || minor SQUASHFS_MINOR) {
+ ERROR(Major/Minor mismatch, trying to mount newer %d.%d
+ filesystem\n, major, minor);
+ ERROR(Please update your kernel\n);
+ return 0;
+ }
+
+ if (compression != ZLIB_COMPRESSION)
+ return 0;
+
+ return 1;
+}
+
+
+static int squashfs_fill_super(struct super_block *s, void *data, int silent)
+{
+ struct squashfs_sb_info *msblk;
+ struct squashfs_super_block *sblk = NULL;
+ char b[BDEVNAME_SIZE];
+ struct inode *root;
+ long long root_inode;
+ unsigned short flags;
+ unsigned int fragments;
+ long long lookup_table_start;
+ int res;
+
+ TRACE(Entered squashfs_fill_superblock\n);
+
+ s-s_fs_info = kzalloc(sizeof(*msblk), GFP_KERNEL);
+ if (s-s_fs_info == NULL) {
+ ERROR(Failed to allocate squashfs_sb_info\n);
+ goto failure2;
+ }
+ msblk = s-s_fs_info;
+
+ msblk-stream.workspace = vmalloc(zlib_inflate_workspacesize());
+ if (msblk-stream.workspace == NULL) {
+ ERROR(Failed to allocate zlib workspace\n);
+ goto failure;
+ }
+
+ sblk = kzalloc(sizeof(*sblk), GFP_KERNEL);
+ if (sblk == NULL) {
+ ERROR(Failed to allocate squashfs_super_block\n);
+ goto failure;
+ }
+
+ msblk-devblksize = sb_min_blocksize(s, BLOCK_SIZE);
+ msblk-devblksize_log2 = ffz(~msblk-devblksize);
+
+ mutex_init(msblk-read_data_mutex);
+ mutex_init(msblk-read_page_mutex);
+ mutex_init(msblk-meta_index_mutex);
+
+ /*
+* msblk-bytes_used is checked in squashfs_read_data to ensure reads
+* are not beyond filesystem end. But as we're using squashfs_read_data
+* here to read the superblock (including the value of
+* bytes_used) we need to set it to an initial sensible dummy value
+*/
+ msblk-bytes_used = sizeof(*sblk);
+ res = squashfs_read_data(s, sblk, SQUASHFS_START, sizeof(*sblk) |
+ SQUASHFS_COMPRESSED_BIT_BLOCK, NULL, sizeof(*sblk));
+
+ if (res == 0) {
+ ERROR(unable to read squashfs_super_block\n);
+ goto failed_mount;
+ }
+
+ /* Check it is a SQUASHFS superblock */
+ s-s_magic = le32_to_cpu(sblk-s_magic);
+ if (s-s_magic != SQUASHFS_MAGIC) {
+ ERROR(Can't find a SQUASHFS superblock on %s\n,
+ bdevname(s-s_bdev, b));
+ goto failed_mount;
+ }
+
+ /* Check the MAJOR MINOR versions and compression type */
+ if (!supported_squashfs_filesystem
Re: [Squashfs-devel] [patch 0/3] [RFC] zlib crypto module
Geert Uytterhoeven wrote: These patches add a (de)compression module for the zlib format using the crypto API: [1] crypto: Add a zlib crypto module [2] tcrypt: Add a self test for the zlib crypto module [3] squashfs: Switch from zlib/inflate to zlib crypto module The last patch is a proof-of-concept to make SquashFS 3.4 use this new zlib crypto module. This makes it easier to e.g. change the decompression algorithm in SquashFS or to make use of a hardware-accelerated zlib crypto module. It can be extended to other compressed file systems, like e.g. AxFS and CRAMFS. All comments are welcome. Thanks! Moving Squashfs over to the crypto API is a good idea. When the zlib crypto module code is in the mainline kernel moving Squashfs, AxFS and CRAMFS over will be easy. Are you planning to do the necessary work to get this (or a subsequent version following the comments from Herbert Xu) into mainline? Cheers Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 04/10] AXFS: axfs_inode.c
Arnd Bergmann wrote: On Friday 22 August 2008, Phillip Lougher wrote: This looks very nice, but could use some comments about how the data is actually stored on disk. It took me some time to figure out that it actually allows to do tail merging into compressed blocks, which I was about to suggest you implement ;-). Cramfs doesn't have them, and I found that they are the main reason why squashfs compresses better than cramfs, besides the default block size, which you can change on either one. Squashfs has much larger block sizes than cramfs (last time I looked it was limited to 4K blocks), and it compresses the metadata which helps to get better compression. But tail merging (fragments in Squashfs terminology) is obviously a major reason why Squashfs gets good compression. The *default* block size in cramfs is smaller than in squashfs, but they both have user selectable block sizes. I found the impact of compressed metadata to be almost zero. Squashfs stores significantly more metadata than cramfs. Remember cramfs has no support for filesystems ~ 16Mbytes, no inode timestamps, truncates uid/gids, no hard-links, no nlink counts, no hashed directories, no unique inode numbers. If Squashfs didn't compress the metadata it would be significantly larger than cramfs. Cheers Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 06/10] AXFS: axfs_super.c
Arnd Bergmann wrote: On Friday 22 August 2008, Jared Hulbert wrote: This implies for block devices that the entire filesystem metadata has to be cached in RAM. This severely limits the size of AXFS filesystems when using block devices, or the else memory usage will be excessive. This is where 64bit squashfs could be a better fit. Is this the only place where squashfs has a significant advantage? If so, you might want to change it in axfs eventually to make the decision easier for users ;-) As you asked here's the list. 1. Support for 4GB filesystems. In theory 2^64 bytes. 2. Compressed metadata 3. Inode timestamps 4. Hard-link support, and correct nlink counts 5. Sparse file support 6. Support for . .. in readdir 7. Indexed directories for fast lookup 8. NFS exporting 9. No need to cache entire metadata in memory Squashfs has been optimised for block-based rotating media like hard disks, CDROMS. AXFS has been optimised for flash based media. Squashfs will outperform AXFS on rotating media, AXFS will outperform Squashfs on flash based media. Squashfs and AXFS should be seen as complementary filesystems, and there should be room in the Linux kernel for both. I don't see what your problem is here. I think AXFS is an extremely good filesystem and should be merged. But I don't see why this should lead to more Squashfs bashing. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 06/10] AXFS: axfs_super.c
Jared Hulbert wrote:
+static void axfs_free_region(struct axfs_super *sbi,
+struct axfs_region_desc *region)
+{
+ if (!region)
+ return;
+
+ if (AXFS_IS_REGION_XIP(sbi, region))
+ return;
+
+ if (region-virt_addr)
+ vfree(region-virt_addr);
+}
+
No need to do
if(xxx)
vfree(xxx)
vfree/kfree can cope with NULL pointers.
+static void axfs_put_sbi(struct axfs_super *sbi)
+{
+ axfs_free_region(sbi, sbi-uids);
+ axfs_free_region(sbi, sbi-gids);
+
+ if (sbi-second_dev)
+ kfree(sbi-second_dev);
+
+ if (sbi-cblock_buffer[0])
+ vfree(sbi-cblock_buffer[0]);
+ if (sbi-cblock_buffer[1])
+ vfree(sbi-cblock_buffer[1]);
+
Again, just kfree(xxx)/vfree(xxx)
+
+static int axfs_copy_mem(struct super_block *sb, void *buf, u64 fsoffset,
+u64 len)
+{
+ struct axfs_super *sbi = AXFS_SB(sb);
+ unsigned long addr;
+
+ addr = sbi-virt_start_addr + (unsigned long)fsoffset;
+ memcpy(buf, (void *)addr, (size_t) len);
+ return 0;
Always returns 0, consider changing to static void
+}
+
+static int axfs_copy_metadata(struct super_block *sb, void *buf, u64 fsoffset,
+ u64 len)
+{
+ struct axfs_super *sbi = AXFS_SB(sb);
+ u64 end = fsoffset + len;
+ u64 a = sbi-mmap_size - fsoffset;
+ u64 b = end - sbi-mmap_size;
+ void *bb = (void *)((unsigned long)buf + (unsigned long)a);
+ int err;
+
+ /* Catches case where sbi is not yet fully initialized. */
+ if ((sbi-magic == 0) (sbi-virt_start_addr != 0))
+ return axfs_copy_mem(sb, buf, fsoffset, len);
+
+ if (fsoffset sbi-mmap_size) {
+ if (end sbi-mmap_size) {
+ err = axfs_copy_metadata(sb, buf, fsoffset, a);
+ if (err)
+ return err;
+ err = axfs_copy_metadata(sb, bb, sbi-mmap_size, b);
+ } else {
+ if (AXFS_IS_OFFSET_MMAPABLE(sbi, fsoffset)) {
+ err = axfs_copy_mem(sb, buf, fsoffset, len);
+ } else if (AXFS_HAS_MTD(sb)) {
+ err = axfs_copy_mtd(sb, buf, fsoffset, len);
+ } else if (AXFS_HAS_BDEV(sb)) {
+ err = axfs_copy_block(sb, buf, fsoffset, len);
+ } else {
+ err = -EINVAL;
Consider initialising err to -EINVAL at declaration time, and get rid of
this else,
+ }
+ }
+ } else {
+ if (AXFS_NODEV(sb)) {
+ err = axfs_copy_mem(sb, buf, fsoffset, len);
+ } else if (AXFS_HAS_BDEV(sb)) {
+ err = axfs_copy_block(sb, buf, fsoffset, len);
+ } else if (AXFS_HAS_MTD(sb)) {
+ err = axfs_copy_mtd(sb, buf, fsoffset, len);
+ } else {
+ err = -EINVAL;
and this one.
+ }
+ }
+ return err;
+}
+
+static int axfs_fill_region_data(struct super_block *sb,
+struct axfs_region_desc *region, int force)
+{
+
+ if (AXFS_IS_REGION_INCORE(region))
+ goto incore;
+
+ if (AXFS_IS_REGION_COMPRESSED(region))
+ goto incore;
+
+ if (AXFS_IS_REGION_XIP(sbi, region)) {
+ if ((end sbi-mmap_size) (force))
+ goto incore;
+ addr = sbi-virt_start_addr;
+ addr += (unsigned long)fsoffset;
+ region-virt_addr = (void *)addr;
+ return 0;
+ }
+
+incore:
+ region-virt_addr = vmalloc(size);
+ if (!region-virt_addr)
+ goto out;
+ vaddr = region-virt_addr;
+
+ if (AXFS_IS_REGION_COMPRESSED(region)) {
+ buff = vmalloc(c_size);
+ if (!buff)
+ goto out;
+ axfs_copy_metadata(sb, buff, fsoffset, c_size);
+ err = axfs_uncompress_block(vaddr, size, buff, c_size);
+ if (!err)
+ goto out;
+ vfree(buff);
+ } else {
+ axfs_copy_metadata(sb, vaddr, fsoffset, size);
+ }
+
+ return 0;
From this it would appear that if the region data can't be mapped XIP
(i.e. it is compressed or on a block device), it is cached in its
entirety in RAM?
This implies for block devices that the entire filesystem metadata has
to be cached in RAM. This severely limits the size of AXFS filesystems
when using block devices, or the else memory usage will be excessive.
+
+out:
+ if (buff)
+ vfree(buff);
+ if (region-virt_addr)
+ vfree(region-virt_addr);
+ return err;
Re: [PATCH 04/10] AXFS: axfs_inode.c
Jared Hulbert wrote: The memcpy in question copies a c_node to the page. The len is either the max length of a c_node and size of the buffer I'm copying to (PAGE_CACHE_SIZE) or it is the difference between the beginning of the c_node in the c_block and the end of the c_block, whichever is smaller. The confusion is probably because of the fact that this copies extra crap to the page for tails. Ah yes, that's where I got confused :) Glad to see AXFS uses tail packing. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Adding a new platform
Geert Uytterhoeven wrote: On Thu, 21 Aug 2008, Charles Manning wrote: Luckily APIs for drivers (the most common stuff that people work on) don't change that much, and the interfaces are reasonably clear. If you want some hell then try working on file systems :-). Really? So how come so few changes are needed to keep squashfs working? It only uses a relatively small subset of the VFS, and goes nowhere near the MTD subsystem. Recently dusting off an MTD patch I wrote for Squashfs a couple of years ago, I found it had to be completely rewritten for modern kernels. Having said that the bit of the VFS Squashfs is interested in has changed in each of 2.6.23, 2.6.24, 2.6.25, 2.6.26 and now 2.6.27, and so I wouldn't say it was immune anyway. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
