Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Jared, nice work! I've also read your paper from the linux symposium (http://ols.fedoraproject.org/OLS/Reprints-2008/hulbert-reprint.pdf) A few questions: - how does this benchmark compared to cramfs and squashfs in a NAND-only system (or is it just not a good plan to use this with NAND-only (of course I won't get XIP with NAND, I understand that) - would axfs be suitable as a filesystem on a ram disk? Background for the last question is that if you do not have the memory to retain all pages uncompressed (as you would with ramfs), this could be a nice intermediate format. Furthermore compared to ramfs, a filesystem on a ramdisk does not need the initialisation during startup (decompressing the cpio file, creating the files, copying the data), so when it comes to boot times a filesystem on a ramdisk (e.g. axfs) could be a better choice. Appreciate your feedback. Frans. -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 05/10] AXFS: axfs_profiling.c
Jared Hulbert wrote: Profiling is a fault instrumentation and /proc formating system. This is used to get an accurate picture of what the pages are actually used. Using this info the image can be optimized for XIP Exporting profiling data for a file system in another file system (/proc) seems not very straigtforward to me. I think it is worth considering to export this information via the same mount point. -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Adding a new platform
On Thu, 21 Aug 2008, Charles Manning wrote: On Wednesday 20 August 2008 17:15:01 vb wrote: On Tue, Aug 19, 2008 at 9:29 PM, Paul Gortmaker [EMAIL PROTECTED] wrote: On Tue, Aug 19, 2008 at 11:57 PM, vb [EMAIL PROTECTED] wrote: so, say a developer submits a proprietary driver and it gets accepted. Doesn't happen. By design. If the driver is proprietary then it is presumably not meant for open distribution, and hence not compatible with GPL and widespread distribution into 100,000 public git repositories. So it won't get submitted and it won't get accepted. I guess 'proprietary' is not the right term then, how do you call a driver which is not a secret and not a problem to release, but controls some hardware present in only in certain devices of a certain company. Would such a driver be accepted? Wouldn't such a driver get stale after a few kernel releases? In-tree is no silver bullet. When people modify internal APIs they will likely fix anything that breaks compilation. However many things are more subtle than that and it is very easy to end up with a driver or other code that compiles but does not work properly. My experience with Linux/m68k tells me that +95% of the breakage is visible by compiler warnings and errors. If it still compiles, it still works ;-) Luckily APIs for drivers (the most common stuff that people work on) don't change that much, and the interfaces are reasonably clear. If you want some hell then try working on file systems :-). Really? So how come so few changes are needed to keep squashfs working? With kind regards, Geert Uytterhoeven Software Architect Sony Techsoft Centre Europe The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium Phone:+32 (0)2 700 8453 Fax: +32 (0)2 700 8622 E-mail: [EMAIL PROTECTED] Internet: http://www.sony-europe.com/ A division of Sony Europe (Belgium) N.V. VAT BE 0413.825.160 · RPR Brussels Fortis · BIC GEBABEBB · IBAN BE41293037680010
Re: [PATCH 10/10] AXFS: axfs_uncompress.c
On Wed, 20 Aug 2008, Jared Hulbert wrote:
Handles the decompression for axfs, modeled after fs/cramfs/uncompress.c.
Signed-off-by: Jared Hulbert [EMAIL PROTECTED]
---
diff --git a/fs/axfs/axfs_uncompress.c b/fs/axfs/axfs_uncompress.c
new file mode 100644
index 000..b7a2060
--- /dev/null
+++ b/fs/axfs/axfs_uncompress.c
@@ -0,0 +1,97 @@
+/*
+ * Advanced XIP File System for Linux - AXFS
+ * Readonly, compressed, and XIP filesystem for Linux systems big and small
+ *
+ * Modified in 2006 by Eric Anderson
+ * from the cramfs sources fs/cramfs/uncompress.c
+ *
+ * (C) Copyright 1999 Linus Torvalds
+ *
+ * axfs_uncompress.c -
+ * axfs interfaces to the uncompression library. There's really just
+ * three entrypoints:
+ *
+ * - axfs_uncompress_init() - called to initialize the thing.
+ * - axfs_uncompress_exit() - tell me when you're done
+ * - axfs_uncompress_block() - uncompress a block.
+ *
+ * NOTE NOTE NOTE! The uncompression is entirely single-threaded. We
+ * only have one stream, and we'll initialize it only once even if it
+ * then is used by multiple filesystems.
+ *
+ */
+
+#include linux/errno.h
+#include linux/vmalloc.h
+#include linux/zlib.h
+#include linux/mutex.h
+
+static z_stream stream;
+static int initialized;
+static struct mutex axfs_uncmp_mutex;
Use DEFINE_MUTEX and drop the mutex_init() down in the init function.
+
+int axfs_uncompress_block(void *dst, int dstlen, void *src, int srclen)
+{
+ int err;
+ int out;
+
+ mutex_lock(axfs_uncmp_mutex);
+
+ stream.next_in = src;
+ stream.avail_in = srclen;
+
+ stream.next_out = dst;
+ stream.avail_out = dstlen;
+
+ err = zlib_inflateReset(stream);
+ if (err != Z_OK) {
+ printk(KERN_ERR zlib_inflateReset error %d\n, err);
+ zlib_inflateEnd(stream);
+ zlib_inflateInit(stream);
+ }
+
+ err = zlib_inflate(stream, Z_FINISH);
+ if (err != Z_STREAM_END)
+ goto err;
+
+ out = stream.total_out;
+
+ mutex_unlock(axfs_uncmp_mutex);
+
+ return out;
+
+err:
+
+ mutex_unlock(axfs_uncmp_mutex);
+
+ printk(KERN_ERR Error %d while decompressing!\n, err);
+ printk(KERN_ERR %p(%d)-%p(%d)\n, src, srclen, dst, dstlen);
+ return 0;
+}
+
+int axfs_uncompress_init(void)
+{
+ if (!initialized++) {
+
+ mutex_init(axfs_uncmp_mutex);
+
+ stream.workspace = vmalloc(zlib_inflate_workspacesize());
+ if (!stream.workspace) {
+ initialized = 0;
+ return -ENOMEM;
+ }
+ stream.next_in = NULL;
+ stream.avail_in = 0;
+ zlib_inflateInit(stream);
+ }
+ return 0;
+}
+
+int axfs_uncompress_exit(void)
+{
+ if (!--initialized) {
+ zlib_inflateEnd(stream);
+ vfree(stream.workspace);
+ }
+ return 0;
+}
axfs_uncompress_init() and axfs_uncompress_exit() are only called during
init and exit of the module, no need for the initialized variable and the
functions can be annotated with __init and __exit.
--
To unsubscribe from this list: send the line unsubscribe linux-embedded in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 06/10] AXFS: axfs_super.c
On Wed, 20 Aug 2008, Jared Hulbert wrote:
The many different devices AXFS can mount to and the various
dual device mounting schemes are supported here.
Signed-off-by: Jared Hulbert [EMAIL PROTECTED]
---
diff --git a/fs/axfs/axfs_super.c b/fs/axfs/axfs_super.c
new file mode 100644
index 000..5efab38
--- /dev/null
+++ b/fs/axfs/axfs_super.c
@@ -0,0 +1,864 @@
+/*
+ * Advanced XIP File System for Linux - AXFS
+ * Readonly, compressed, and XIP filesystem for Linux systems big and small
+ *
+ * Copyright(c) 2008 Numonyx
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * Authors:
+ * Eric Anderson
+ * Jared Hulbert [EMAIL PROTECTED]
+ * Sujaya Srinivasan
+ * Justin Treon
+ *
+ * More info and current contacts at http://axfs.sourceforge.net
+ *
+ * axfs_super.c -
+ * Contains the core code used to mount the fs.
+ *
+ */
+
+#include linux/axfs.h
+#include linux/fs.h
+#include linux/vmalloc.h
+#include linux/parser.h
+#include linux/statfs.h
+#include linux/module.h
+#include linux/mount.h
+#include linux/mtd/mtd.h
+
+/ Function Declarations /
+static struct super_operations axfs_sops;
+static struct axfs_super *axfs_get_sbi(void);
+static void axfs_put_sbi(struct axfs_super *);
If you reorder the code slightly you can avoid these static forward
declarations.
+/* functions in other axfs files ***/
+int axfs_get_sb_bdev(struct file_system_type *, int, const char *,
+ struct axfs_super *, struct vfsmount *, int *);
+void axfs_kill_block_super(struct super_block *);
+int axfs_copy_block(struct super_block *, void *, u64, u64);
+int axfs_is_dev_bdev(char *);
+int axfs_map_mtd(struct super_block *);
+void axfs_unmap_mtd(struct super_block *);
+int axfs_copy_mtd(struct super_block *, void *, u64, u64);
+int axfs_get_sb_mtd(struct file_system_type *, int, const char *,
+ struct axfs_super *, struct vfsmount *, int *);
+int axfs_is_dev_mtd(char *, int *);
+void axfs_kill_mtd_super(struct super_block *);
+struct inode *axfs_create_vfs_inode(struct super_block *, int);
+int axfs_get_uml_address(char *, unsigned long *, unsigned long *);
+int axfs_init_profiling(struct axfs_super *);
+int axfs_shutdown_profiling(struct axfs_super *);
+void axfs_profiling_add(struct axfs_super *, unsigned long, unsigned int);
+struct inode *axfs_create_vfs_inode(struct super_block *, int);
+/**/
These should probably be in axfs.h.
+static int axfs_fill_region_descriptors(struct super_block *sb,
+ struct axfs_super_onmedia *sbo)
+{
+ struct axfs_super *sbi = AXFS_SB(sb);
+ struct axfs_region_desc_onmedia *out;
+
+ out = kmalloc(sizeof(*out), GFP_KERNEL);
+ if (!out)
+ return -ENOMEM;
+ memset(out, 0, sizeof(*out));
kzalloc()
+
+ axfs_fill_region_desc(sb, out, sbo-strings, sbi-strings);
+ axfs_fill_region_desc(sb, out, sbo-xip, sbi-xip);
+ axfs_fill_region_desc(sb, out, sbo-compressed, sbi-compressed);
+ axfs_fill_region_desc(sb, out, sbo-byte_aligned, sbi-byte_aligned);
+ axfs_fill_region_desc(sb, out, sbo-node_type, sbi-node_type);
+ axfs_fill_region_desc(sb, out, sbo-node_index, sbi-node_index);
+ axfs_fill_region_desc(sb, out, sbo-cnode_offset, sbi-cnode_offset);
+ axfs_fill_region_desc(sb, out, sbo-cnode_index, sbi-cnode_index);
+ axfs_fill_region_desc(sb, out, sbo-banode_offset, sbi-banode_offset);
+ axfs_fill_region_desc(sb, out, sbo-cblock_offset, sbi-cblock_offset);
+ axfs_fill_region_desc(sb, out, sbo-inode_file_size,
+ sbi-inode_file_size);
+ axfs_fill_region_desc(sb, out, sbo-inode_name_offset,
+ sbi-inode_name_offset);
+ axfs_fill_region_desc(sb, out, sbo-inode_num_entries,
+ sbi-inode_num_entries);
+ axfs_fill_region_desc(sb, out, sbo-inode_mode_index,
+ sbi-inode_mode_index);
+ axfs_fill_region_desc(sb, out, sbo-inode_array_index,
+ sbi-inode_array_index);
+ axfs_fill_region_desc(sb, out, sbo-modes, sbi-modes);
+ axfs_fill_region_desc(sb, out, sbo-uids, sbi-uids);
+ axfs_fill_region_desc(sb, out, sbo-gids, sbi-gids);
+
+ kfree(out);
+
+ return 0;
+}
+int axfs_set_compression_type(struct axfs_super *sbi)
static
And this function actually doesn't set anything, check might be a better
name.
+{
+ if (sbi-compression_type != ZLIB)
+ return -EINVAL;
+
+ return 0;
+}
+/* Read the last four bytes of the volume and make sure the AXFS magic is
+
Re: [PATCH 05/10] AXFS: axfs_profiling.c
On Thu, 2008-08-21 at 10:44 +0200, Carsten Otte wrote: Jared Hulbert wrote: Profiling is a fault instrumentation and /proc formating system. This is used to get an accurate picture of what the pages are actually used. Using this info the image can be optimized for XIP Exporting profiling data for a file system in another file system (/proc) seems not very straigtforward to me. I think it is worth considering to export this information via the same mount point. I would have said sysfs, rather than 'the same mount point'. -- David WoodhouseOpen Source Technology Centre [EMAIL PROTECTED] Intel Corporation -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Jared Hulbert wrote: I'd like to get a first round of review on my AXFS filesystem. I like the general approach of it. It's much more flexible than the ext2 extension I've done, and the possibility to select XIP vs. compression per page is really really neat. I can imagine that people will prefer this over the ext2 implementation on s390. It is unclear to me how the secondary block device thing is supposed to work. Could you elaborate a bit on that? -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
On Thursday 21 August 2008 20:25, Carsten Otte wrote: Jared Hulbert wrote: I'd like to get a first round of review on my AXFS filesystem. I like the general approach of it. It's much more flexible than the ext2 extension I've done, and the possibility to select XIP vs. compression per page is really really neat. I can imagine that people will prefer this over the ext2 implementation on s390. It is unclear to me how the secondary block device thing is supposed to work. Could you elaborate a bit on that? Agreed. I haven't had a good look through it yet, but at a glance it looks pretty neat. The VM side of things looks pretty reasonable (I fear XIP faulting might have another race or two, but that's a core mm issue rather than filesystem specific). -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Jared Hulbert wrote: The biggest improvement is in the way AXFS allows for each page to be XIP or not. First, a user collects information about which pages are accessed on a compressed image for each mmap()ed region from /proc/axfs/volume0. That 'profile' is used as an input to the image builder. The resulting image has only the relevant pages uncompressed and XIP. The result is smaller memory sizes and faster launches. Sounds great, really nice idea. How does it fare with no MMU? Can the profiler and image builder lay out the XIP pages in such a way that no-MMU mmaps can map those regions? No complaint if not, it would be a nice bonus though. -- Jamie -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 02/10] AXFS: Kconfig and Makefiles
On Thursday 21 August 2008, Jared Hulbert wrote: The Kconfig edits and Makefiles required for AXFS. Signed-off-by: Jared Hulbert [EMAIL PROTECTED] If you split out this patch separate from the files, please make it the *last* patch so that you cannot get build errors during a later git-bisect through the middle of your series. Arnd -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 04/10] AXFS: axfs_inode.c
On Thursday 21 August 2008, Jared Hulbert wrote: +/* functions in other axfs files **/ +int axfs_get_sb(struct file_system_type *, int, const char *, void *, + struct vfsmount *); +void axfs_kill_super(struct super_block *); +void axfs_profiling_add(struct axfs_super *, unsigned long, unsigned int); +int axfs_copy_mtd(struct super_block *, void *, u64, u64); +int axfs_copy_block(struct super_block *, void *, u64, u64); *Never* put extern declarations into a .c file, that's what headers are for. If you ever change the definition, the compiler doesn't get a chance to warn you otherwise. +/**/ +static int axfs_readdir(struct file *, void *, filldir_t); +static int axfs_mmap(struct file *, struct vm_area_struct *); +static ssize_t axfs_file_read(struct file *, char __user *, size_t, loff_t *); +static int axfs_readpage(struct file *, struct page *); +static int axfs_fault(struct vm_area_struct *, struct vm_fault *); +static struct dentry *axfs_lookup(struct inode *, struct dentry *, + struct nameidata *); +static int axfs_get_xip_mem(struct address_space *, pgoff_t, int, void **, + unsigned long *); For style reasons, also please don't put static forward declarations anywhere, but define the functions in the right order so you don't need them. Arnd -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 05/10] AXFS: axfs_profiling.c
On Thursday 21 August 2008, David Woodhouse wrote: On Thu, 2008-08-21 at 10:44 +0200, Carsten Otte wrote: Exporting profiling data for a file system in another file system (/proc) seems not very straigtforward to me. I think it is worth considering to export this information via the same mount point. I would have said sysfs, rather than 'the same mount point'. Let me throw in debugfs as my preferred option. sysfs is for stable interfaces, while profiling generally fits into the debugging category. Arnd -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 04/10] AXFS: axfs_inode.c
On Thursday 21 August 2008, Jared Hulbert wrote:
+ array_index = AXFS_GET_INODE_ARRAY_INDEX(sbi, ino_number);
+ array_index += page-index;
+
+ node_index = AXFS_GET_NODE_INDEX(sbi, array_index);
+ node_type = AXFS_GET_NODE_TYPE(sbi, array_index);
+
+ if (node_type == Compressed) {
+ /* node is in compessed region */
+ cnode_offset = AXFS_GET_CNODE_OFFSET(sbi, node_index);
+ cnode_index = AXFS_GET_CNODE_INDEX(sbi, node_index);
+ down_write(sbi-lock);
+ if (cnode_index != sbi-current_cnode_index) {
+ /* uncompress only necessary if different cblock */
+ ofs = AXFS_GET_CBLOCK_OFFSET(sbi, cnode_index);
+ len = AXFS_GET_CBLOCK_OFFSET(sbi, cnode_index + 1);
+ len -= ofs;
+ axfs_copy_data(sb, cblk1, (sbi-compressed), ofs,
len);
+ axfs_uncompress_block(cblk0, cblk_size, cblk1, len);
+ sbi-current_cnode_index = cnode_index;
+ }
+ downgrade_write(sbi-lock);
+ max_len = cblk_size - cnode_offset;
+ len = max_len PAGE_CACHE_SIZE ? PAGE_CACHE_SIZE : max_len;
+ src = (void *)((unsigned long)cblk0 + cnode_offset);
+ memcpy(pgdata, src, len);
+ up_read(sbi-lock);
This looks very nice, but could use some comments about how the data is
actually stored on disk. It took me some time to figure out that it actually
allows to do tail merging into compressed blocks, which I was about to suggest
you implement ;-). Cramfs doesn't have them, and I found that they are the
main reason why squashfs compresses better than cramfs, besides the default
block size, which you can change on either one.
Have you seen any benefit of the rwsem over a simple mutex? I would guess
that you can never even get into the situation where you get concurrent
readers since I haven't found a single down_read() in your code, only
downgrade_write().
Arnd
Re: [PATCH 10/10] AXFS: axfs_uncompress.c
On Thu, 21 Aug 2008, Artem Bityutskiy wrote:
On Wed, 2008-08-20 at 22:46 -0700, Jared Hulbert wrote:
+ err = zlib_inflateReset(stream);
+ if (err != Z_OK) {
+ printk(KERN_ERR zlib_inflateReset error %d\n, err);
+ zlib_inflateEnd(stream);
+ zlib_inflateInit(stream);
+ }
just FYI, are you aware that LZO which is also present in the kernel is
much faster on decompress than zlib, while its compression is only
slightly worse?
If you want support for multiple decompression algorithms, you can switch
from using zlib_inflate*() directly to calling zlib through the crypto API.
Then you can call crypto_alloc_comp() with the correct decompression algorithm
name.
For squashfs, I had to modify only ca. 40 lines of code.
You do need a new zlib crypto module, as the existing deflate crypto module
uses the raw deflate format instead of the zlib format, and has some parameters
tuned for its use in IPSec.
I hope to have some patches ready next week...
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone:+32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: [EMAIL PROTECTED]
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
Re: [PATCH 03/10] AXFS: axfs.h
On Wed, 2008-08-20 at 22:45 -0700, Jared Hulbert wrote: +#define AXFS_GET_BYTETABLE_VAL(desc,index) \ + axfs_bytetable_stitch(((struct axfs_region_desc)(desc)).table_byte_depth,\ + (u8 *)((struct axfs_region_desc)(desc)).virt_addr, index) + +#define AXFS_GET_NODE_TYPE(sbi,node_index) \ + AXFS_GET_BYTETABLE_VAL(((struct axfs_super *)(sbi))-node_type,\ + (node_index)) + +#define AXFS_GET_NODE_INDEX(sbi,node__index) \ + AXFS_GET_BYTETABLE_VAL(((struct axfs_super *)(sbi))-node_index,\ + (node__index)) + I think it would be much cleaner to do all these similar macro's as static inline functions. Daniel -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
I like the general approach of it. It's much more flexible than the ext2 extension I've done, and the possibility to select XIP vs. compression per page is really really neat. I can imagine that people will prefer this over the ext2 implementation on s390. It is unclear to me how the secondary block device thing is supposed to work. Could you elaborate a bit on that? First off we don't yet support direct_access(), but I am planning on that soon. Sure. For a system that has say a NOR Flash and a NAND or a embedded MMC, one can split a filesystem image such that only the XIP parts of the image are on the NOR while the compressed bits are on the NAND / eMMC. The NOR part is accessed as directly addressable memory, while the NAND would use mtd-read() and the eMMC would use block device access API's. In this case I would call this NAND or eMMC the secondary device because the primary device is the NOR. Assuming my NOR was at /dev//mtd2 and my NAND at /dev/mtd5. I would call the following to mount such a system: mount -t axfs -o second_dev=/dev/mtd5 /dev/mtd2 /mnt/axfs -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 10/10] AXFS: axfs_uncompress.c
If you want support for multiple decompression algorithms, you can switch from using zlib_inflate*() directly to calling zlib through the crypto API. Then you can call crypto_alloc_comp() with the correct decompression algorithm name. For squashfs, I had to modify only ca. 40 lines of code. I definately want to support multiple formats. I have a flag in the superblock all ready to go for that. One problem. I'm not sure how to do it. Can you point us to some reference code? -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
Hi Jared, Jared Hulbert wrote: How does it fare with no MMU? Can the profiler and image builder lay out the XIP pages in such a way that no-MMU mmaps can map those regions? No complaint if not, it would be a nice bonus though. Sorry. I don't believe it will work on no-MMU as is. That said you _could_ tweak the mkfs tool to lay mmap()'ed regions down contiguously but then if you mmap() an unprofiled region, well that would be bad. I suppose you could make axfs_mmap smart enough to handle that. I guess the cleanest way would be to just make files lay down contiguously, you lose some of the space saving but it would work. That would be enough I think. If you could manually select which files are contiguous-and-uncompressed that would be useful for some too here. I'm not plannin to get to this anytime soon. But I'd be willing merge patches. Can anybody convince me offline that working on no-MMU this makes financial sense for my employer? This is getting to be a common question. How many noMMU users are out there and why are you so interested? One of those unknown factors, how many are there? Who knows, pretty much impossible to tell. One thing for sure is that many people who do non-MMU setups are interested in XIP to get the space savings. These are very often small devices with very constrained RAM and flash. (For whatever it is worth single NOR flash only boards are common in these smaller form factors :-) Regards Greg Greg Ungerer -- Chief Software Dude EMAIL: [EMAIL PROTECTED] Secure Computing CorporationPHONE: +61 7 3435 2888 825 Stanley St, FAX: +61 7 3891 3630 Woolloongabba, QLD, 4102, Australia WEB: http://www.SnapGear.com -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 06/10] AXFS: axfs_super.c
Jared Hulbert wrote:
+static void axfs_free_region(struct axfs_super *sbi,
+struct axfs_region_desc *region)
+{
+ if (!region)
+ return;
+
+ if (AXFS_IS_REGION_XIP(sbi, region))
+ return;
+
+ if (region-virt_addr)
+ vfree(region-virt_addr);
+}
+
No need to do
if(xxx)
vfree(xxx)
vfree/kfree can cope with NULL pointers.
+static void axfs_put_sbi(struct axfs_super *sbi)
+{
+ axfs_free_region(sbi, sbi-uids);
+ axfs_free_region(sbi, sbi-gids);
+
+ if (sbi-second_dev)
+ kfree(sbi-second_dev);
+
+ if (sbi-cblock_buffer[0])
+ vfree(sbi-cblock_buffer[0]);
+ if (sbi-cblock_buffer[1])
+ vfree(sbi-cblock_buffer[1]);
+
Again, just kfree(xxx)/vfree(xxx)
+
+static int axfs_copy_mem(struct super_block *sb, void *buf, u64 fsoffset,
+u64 len)
+{
+ struct axfs_super *sbi = AXFS_SB(sb);
+ unsigned long addr;
+
+ addr = sbi-virt_start_addr + (unsigned long)fsoffset;
+ memcpy(buf, (void *)addr, (size_t) len);
+ return 0;
Always returns 0, consider changing to static void
+}
+
+static int axfs_copy_metadata(struct super_block *sb, void *buf, u64 fsoffset,
+ u64 len)
+{
+ struct axfs_super *sbi = AXFS_SB(sb);
+ u64 end = fsoffset + len;
+ u64 a = sbi-mmap_size - fsoffset;
+ u64 b = end - sbi-mmap_size;
+ void *bb = (void *)((unsigned long)buf + (unsigned long)a);
+ int err;
+
+ /* Catches case where sbi is not yet fully initialized. */
+ if ((sbi-magic == 0) (sbi-virt_start_addr != 0))
+ return axfs_copy_mem(sb, buf, fsoffset, len);
+
+ if (fsoffset sbi-mmap_size) {
+ if (end sbi-mmap_size) {
+ err = axfs_copy_metadata(sb, buf, fsoffset, a);
+ if (err)
+ return err;
+ err = axfs_copy_metadata(sb, bb, sbi-mmap_size, b);
+ } else {
+ if (AXFS_IS_OFFSET_MMAPABLE(sbi, fsoffset)) {
+ err = axfs_copy_mem(sb, buf, fsoffset, len);
+ } else if (AXFS_HAS_MTD(sb)) {
+ err = axfs_copy_mtd(sb, buf, fsoffset, len);
+ } else if (AXFS_HAS_BDEV(sb)) {
+ err = axfs_copy_block(sb, buf, fsoffset, len);
+ } else {
+ err = -EINVAL;
Consider initialising err to -EINVAL at declaration time, and get rid of
this else,
+ }
+ }
+ } else {
+ if (AXFS_NODEV(sb)) {
+ err = axfs_copy_mem(sb, buf, fsoffset, len);
+ } else if (AXFS_HAS_BDEV(sb)) {
+ err = axfs_copy_block(sb, buf, fsoffset, len);
+ } else if (AXFS_HAS_MTD(sb)) {
+ err = axfs_copy_mtd(sb, buf, fsoffset, len);
+ } else {
+ err = -EINVAL;
and this one.
+ }
+ }
+ return err;
+}
+
+static int axfs_fill_region_data(struct super_block *sb,
+struct axfs_region_desc *region, int force)
+{
+
+ if (AXFS_IS_REGION_INCORE(region))
+ goto incore;
+
+ if (AXFS_IS_REGION_COMPRESSED(region))
+ goto incore;
+
+ if (AXFS_IS_REGION_XIP(sbi, region)) {
+ if ((end sbi-mmap_size) (force))
+ goto incore;
+ addr = sbi-virt_start_addr;
+ addr += (unsigned long)fsoffset;
+ region-virt_addr = (void *)addr;
+ return 0;
+ }
+
+incore:
+ region-virt_addr = vmalloc(size);
+ if (!region-virt_addr)
+ goto out;
+ vaddr = region-virt_addr;
+
+ if (AXFS_IS_REGION_COMPRESSED(region)) {
+ buff = vmalloc(c_size);
+ if (!buff)
+ goto out;
+ axfs_copy_metadata(sb, buff, fsoffset, c_size);
+ err = axfs_uncompress_block(vaddr, size, buff, c_size);
+ if (!err)
+ goto out;
+ vfree(buff);
+ } else {
+ axfs_copy_metadata(sb, vaddr, fsoffset, size);
+ }
+
+ return 0;
From this it would appear that if the region data can't be mapped XIP
(i.e. it is compressed or on a block device), it is cached in its
entirety in RAM?
This implies for block devices that the entire filesystem metadata has
to be cached in RAM. This severely limits the size of AXFS filesystems
when using block devices, or the else memory usage will be excessive.
+
+out:
+ if (buff)
+ vfree(buff);
+ if (region-virt_addr)
+ vfree(region-virt_addr);
+ return err;
Re: kernel dump solutions
On Wed, Aug 13, 2008 at 3:48 PM, Josh Boyer [EMAIL PROTECTED] wrote: Hi All, I'm curious what people are using for a kernel dump solution on embedded Create a region of memory that pesist across resets. Put kernel panic info, exceptiions and scheduling events into said memory. Profit! If only more manufactures would support this in the bios... Intel and BIOS writter folks are you listening? -- // Randy -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 00/10] AXFS: Advanced XIP filesystem
That would be enough I think. If you could manually select which files are contiguous-and-uncompressed that would be useful for some too here. So If you don't have an MMU when do you call -fault? Does the noMMU code just loop through -fault()ing all the pages in an mmap()? One thing for sure is that many people who do non-MMU setups are interested in XIP to get the space savings. These are very often small devices with very constrained RAM and flash. (For whatever it is worth single NOR flash only boards are common in these smaller form factors :-) True. -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel dump solutions
On Thu, 2008-08-21 at 21:53 -0400, Randy MacLeod wrote: On Wed, Aug 13, 2008 at 3:48 PM, Josh Boyer [EMAIL PROTECTED] wrote: Hi All, I'm curious what people are using for a kernel dump solution on embedded Create a region of memory that pesist across resets. Put kernel panic info, exceptiions and scheduling events into said memory. Profit! I've done that many times. It works well enough, but it's not a dump. I've actually used some dumps to debug some rather difficult kernel problems in the past, but there are increasingly fewer options these days. If only more manufactures would support this in the bios... Intel and BIOS writter folks are you listening? BIOS... yeah... I don't have a BIOS. That's the best way to deal with that. josh -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 04/10] AXFS: axfs_inode.c
Squashfs has much larger block sizes than cramfs (last time I looked it was limited to 4K blocks), and it compresses the metadata which helps to get better compression. But tail merging (fragments in Squashfs terminology) is obviously a major reason why Squashfs gets good compression. The AXFS code is rather obscure but it doesn't look to me that it does tail merging. The following code wouldn't work if the block in question was a tail contained in a larger block. It assumes the block extends to the end of the compressed block (cblk_size - cnode_offset). A c_block is the unit that gets compressed. It can contain multiple c_nodes. The c_block can be PAGE_SIZE to 4GB in size, in theory :) The c_nodes can be 1B to PAGE_SIZE. in any alignment. I pack many tails as c_nodes in a c_block. + max_len = cblk_size - cnode_offset; + len = max_len PAGE_CACHE_SIZE ? PAGE_CACHE_SIZE : max_len; + src = (void *)((unsigned long)cblk0 + cnode_offset); + memcpy(pgdata, src, len); Perhaps the AXFS authors could clarify this? The memcpy in question copies a c_node to the page. The len is either the max length of a c_node and size of the buffer I'm copying to (PAGE_CACHE_SIZE) or it is the difference between the beginning of the c_node in the c_block and the end of the c_block, whichever is smaller. The confusion is probably because of the fact that this copies extra crap to the page for tails. -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 04/10] AXFS: axfs_inode.c
I assume compressed blocks can be larger than PAGE_CACHE_SIZE? This suffers from the rather obvious inefficiency that you decompress a big block PAGE_CACHE_SIZE, but only copy one PAGE_CACHE_SIZE page out of it. If multiple files are being read simultaneously (a common occurrence), then each is going to replace your one cached uncompressed block (sbi-current_cnode_index), leading to decompressing the same blocks over and over again on sequential file access. readpage file A, index 1 - decompress block X readpage file B, index 1 - decompress block Y (replaces X) readpage file A, index 2 - repeated decompress of block X (replaces Y) readpage file B, index 2 - repeated decompress of block Y (replaces X) and so on. Yep. Been thinking about optimizing it. So far it hasn't been an issue for my customers. Most fs traffic being on the XIP pages. Once I get a good automated performance test up we'll probably look into something to improve this. -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 04/10] AXFS: axfs_inode.c
Jared Hulbert wrote: The memcpy in question copies a c_node to the page. The len is either the max length of a c_node and size of the buffer I'm copying to (PAGE_CACHE_SIZE) or it is the difference between the beginning of the c_node in the c_block and the end of the c_block, whichever is smaller. The confusion is probably because of the fact that this copies extra crap to the page for tails. Ah yes, that's where I got confused :) Glad to see AXFS uses tail packing. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Adding a new platform
Geert Uytterhoeven wrote: On Thu, 21 Aug 2008, Charles Manning wrote: Luckily APIs for drivers (the most common stuff that people work on) don't change that much, and the interfaces are reasonably clear. If you want some hell then try working on file systems :-). Really? So how come so few changes are needed to keep squashfs working? It only uses a relatively small subset of the VFS, and goes nowhere near the MTD subsystem. Recently dusting off an MTD patch I wrote for Squashfs a couple of years ago, I found it had to be completely rewritten for modern kernels. Having said that the bit of the VFS Squashfs is interested in has changed in each of 2.6.23, 2.6.24, 2.6.25, 2.6.26 and now 2.6.27, and so I wouldn't say it was immune anyway. Phillip -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
