Re: [PATCH] fs: Introducing Lanyard Filesystem
Hi, On Sun, Aug 19, 2012 at 5:08 AM, Dan Luedtke wrote: > + > + /* allocate filesystem private data */ > + fsi = kzalloc(sizeof(*fsi), GFP_KERNEL); > + if (!fsi) > + return -ENOMEM; > + spin_lock_init(&fsi->lock); > + sb->s_fs_info = fsi; > + > + /* set blocksize to minimum size for fetching superblock */ > + if (!sb_set_blocksize(sb, 1 << LANYFS_MIN_BLOCKSIZE)) { > + if (!silent) > + lanyfs_err(sb, "error setting blocksize to %d bytes", > + 1 << LANYFS_MIN_BLOCKSIZE); > + return -EIO; > + } > + > + /* fetch superblock */ > + bh = sb_bread(sb, LANYFS_SUPERBLOCK); > + if (!bh) { > + if (!silent) > + lanyfs_err(sb, "error reading superblock"); > + return -EIO; > + } > + lanysb = (struct lanyfs_sb *) bh->b_data; > + > + /* check magic */ > + if (lanysb->magic != cpu_to_le32(LANYFS_SUPER_MAGIC)) { > + if (!silent) > + lanyfs_info(sb, "bad magic 0x%x", > + lanysb->magic); > + goto exit_invalid; > + } > + sb->s_magic = LANYFS_SUPER_MAGIC; > + > + /* check block type */ > + if (lanysb->type != LANYFS_TYPE_SB) { > + if (!silent) > + lanyfs_err(sb, "bad block type 0x%x", lanysb->type); > + goto exit_invalid; > + } > + > + /* check version */ > + if (lanysb->major > LANYFS_MAJOR_VERSION) { > + if (!silent) > + lanyfs_err(sb, "major version mismatch"); > + goto exit_invalid; > + } > + > + /* check address length */ > + if (lanysb->addrlen < LANYFS_MIN_ADDRLEN || > + lanysb->addrlen > LANYFS_MAX_ADDRLEN) { > + if (!silent) > + lanyfs_err(sb, "unsupported address length"); > + goto exit_invalid; > + } > + fsi->addrlen = lanysb->addrlen; > + > + /* check blocksize */ > + if (lanysb->blocksize < LANYFS_MIN_BLOCKSIZE || > + lanysb->blocksize > LANYFS_MAX_BLOCKSIZE) { > + if (!silent) > + lanyfs_err(sb, "unsupported blocksize"); > + goto exit_invalid; > + } > + fsi->blocksize = lanysb->blocksize; > + > + > + /* release block buffer */ > + brelse(bh); > + > + /* parse mount options */ > + save_mount_options(sb, options); > + err = lanyfs_super_options(sb, (char *) options, silent); > + if (err) > + return err; > + > + /* set blocksize to correct size */ > + if (!sb_set_blocksize(sb, 1 << fsi->blocksize)) { > + if (!silent) > + lanyfs_err(sb, "error setting blocksize to %d bytes", > + 1 << fsi->blocksize); > + return -EIO; > + } > + /* default flags */ > + sb->s_maxbytes = 0x; /* TODO: hm */ > + sb->s_op = &lanyfs_super_operations; > + sb->s_time_gran = 1; > + sb->s_flags = MS_NOSUID | MS_NOATIME | MS_NODIRATIME; > + > + /* make root directory */ > + inode = lanyfs_iget(sb, fsi->rootdir); > + if (!inode) > + return -ENOMEM; > + > + sb->s_root = d_make_root(inode); > + if (!sb->s_root) { > + iput(inode); > + return -ENOMEM; > + } > + return 0; > + > +exit_invalid: > + brelse(bh); > + if (!silent) > + lanyfs_info(sb, "no valid lanyard filesystem found"); > + return -EINVAL; > +} You should free the memory for "fsi" on error. Regards. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
Dan Luedtke writes: > +/** > + * struct lanyfs_opts - mount options > + * @uid: userid of all files and directories > + * @gid: grouid of all files and direcotries > + * @dmask: directory mask > + * @fmask: file mask > + * @discard: issue discard requests on block freeing > + * @flush: force instant writing of changed data > + */ > +struct lanyfs_opts { > + uid_t uid; > + gid_t gid; A small nit. Those above two lines should be: kuid_t uid; kgid_t gid; > + unsigned intdmask; > + unsigned intfmask; > + unsigned intdiscard:1, > + flush:1; > +}; Eric -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sunday 2012-08-19 15:34, Dan Luedtke wrote: >I analyzed about 600k file stored on various removable storage devices. >80 volunteers sent in data about their devices, generated by a program >(windows) and scripts (linux, bsd, osx) I wrote for that purpose. The >data shows that people use more complex filesystems as soon as they are >confronted with problems (mostly the 4GB limit). After that they have >problems getting their data accessed by other systems. I derived from >that, that we need a filesystem that is so simple that even unpopular >operating systems can implement it without having their business plan >explode. Those unpopular OSes do not care about *any other* filesystem at all. The only way to get a filesystem accepted is by making an ISO standard out of it - or something in that direction - *and* use it thoroughly in products, like UDF. Even then, support for new revisions of UDF has been rather slow-coming. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Monday 20 August 2012, Theodore Ts'o wrote: > On Mon, Aug 20, 2012 at 11:12:07AM +0200, Alexander Thomas wrote: > > > > Flash drives are getting faster as well. Copying an 8GB file to/from a > > USB drive is not excruciatingly slow and may be quicker and more > > certain than figuring out how to get a working network connection in > > some random place, if possible at all. If it is some lousy WiFi with > > the base station at a distance, a flash drive will be faster. And > > sometimes people just want to be sure that their data will be at a > > certain place at a certain time without having to rely on a network > > that may go down due to external reasons. Actually, flash drives are not really getting faster as much as you think, at least not any more. The main source for performance improvements over the last few years was from increasing the flash page and block sizes. Writing single bits may get slightly faster over time, but we are writing more of them at the same time now, which leads to significant problems: * The page size is now effectively 16kb on most flash drives, and writing in 4kb alignments as practically all our file systems do (including the lanyfs proposal) means we're actually slower now than we would be with smaller pages, unless you get into the special case of writing large chunks of data. * The erase block size is increasing insanely. We're seeing 12MB and 16MB block size devices now, and our simulations have shown that our file systems that are not aware of this block size are hitting a wall at around 4MB. > I used to think that we would need an IP unencumbered file system, > given issues around TomTom and Microsoft, but these days, given how > quickly Linux has taken over the embedded and mobile landscape[2] for > all but the most tiniest of devices, I don't think that's as important > of an issue, since we can just simply use a native linux file system. > In the time that it would take to get some other new file system > adopted across the industry, it's likely Linux will have enough market > share to perhaps compel the other OS vendors to provide > interoperability solutions. (Just as the BSD folks have implemented > ext2 support; Linux hasn't bothered to implement FFS2 support) There will be patches very soon for a new file system from a major flash vendor that I'm cooperating with. I haven't seen the patches myself yet, but the design is similar to a prototype that was done as a thesis I supervised [1]. I hope that the new implementation is similarly simple to this design, and also able to provide optimum performance on most flash media. Arnd [1] https://wiki.linaro.org/WorkingGroups/Kernel/Specs/flash-file-system-prototype -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
Hi, On Sun, 2012-08-19 at 01:38 +0200, Dan Luedtke wrote: > This patch introduces the Lanyard Filesystem (LanyFS), a filesystem > for highly mobile and removable storage devices. > Did you have any performance comparison of your file system with others? Have you any benchmark results? I think that simplicity can be a valuable thing but performance is a key factor, especially for business guys. I think that maybe compression or/and encryption support can be a valuable feature for such niche of file system that you declared. Efficient compression support is very important feature for embedded solutions. Moreover, using of hardware opportunity in the field of compression or encryption can keep driver code simple. By the way, what about fault-tolerance of your file system? I don't dive deeply in documentation of your file systems. But, I think that for USB sticks or removable storages it is very common situation of sudden switch off. So, it is very important for your file system to be a very tolerant to such use-cases. How can you estimate tolerance of your file system architecture for failure as normal situation? Moreover, I think that simplicity and strong tolerance to file system corruption can be a feature. I mean that if you have simple on-disk layout then, maybe, it is possible to try working in very corrupted environment also. For the end user, from my point of view, possibility to work in the case of file system corruption can be very precious feature. I think that also for your file system such feature as easy recoverability of user information in the case of complete corruption of file system can be very useful thing for an end user. Such easy recoverability can be achieved by means of on-disk layout and file system driver's techniques, I think. So, simplicity and easy recoverability of user data can be a valuable feature also. With the best regards, Vyacheslav Dubeyko. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, 2012-08-19 at 17:04 -0400, Theodore Ts'o wrote: > I also seriously question the niche of people who want to use a thumb > drive to transfer > 4GB files. Try it sometime and see what a painful > user experience it is I don't know if LanyFS will it ever make, and to be honest there are some corporations _not_ interested in its success, but if it makes it, this process might be less painful in the future. At least I have to try, and if we have a better solution in a few years, I'd be happy to drop the idea and go back to watching my movies again. However, I watch this thread and I take all the answers and mails into consideration. You are the experts, and I appreciate you take the time to discuss the general problem of compatibility and LanyFS in particular. I don't want to go to deep into the network vs. removable storage discussion, although it is an interesting one. For some reason this issue made a lot of noise, and I have to concentrate on replying to various mails and modifying a lot of code eventually. Thanks all of you for your comments. Regards Dan -- Dan Luedtke http://www.danrl.de -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, 2012-08-19 at 16:12 +0100, Al Viro wrote: > Conversions *in* *place* are bad. [+explanation] I think I got it now. On Sun, 2012-08-19 at 17:24 +0200, Marco Stornelli wrote: > [vmtruncate, dio_wait, locking, d_delete] Noted! Thanks both of you! Regards Dan -- Dan Luedtke http://www.danrl.de -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Mon, Aug 20, 2012 at 11:12:07AM +0200, Alexander Thomas wrote: > > Flash drives are getting faster as well. Copying an 8GB file to/from a > USB drive is not excruciatingly slow and may be quicker and more > certain than figuring out how to get a working network connection in > some random place, if possible at all. If it is some lousy WiFi with > the base station at a distance, a flash drive will be faster. And > sometimes people just want to be sure that their data will be at a > certain place at a certain time without having to rely on a network > that may go down due to external reasons. OK --- and how many of these situations will you be using such a stripped down operating system that you can't afford to implement ntfs or ext4? Samsung is considering an Android-power point-and-shoot digital camera[1]. And mobile phones are using Android phones where implementation of ext4 is a worked example (and not particularly difficult, either). If this file system had gotten implemented for Arduino first (and someone actually had a worked example of why you would need > 4GB vfiles for an Arduino device --- if I were going to be implementing something with video I'd probably be using a full Linux kernel), the claimed use case might be more compelling. It would also demonstrate that in fact this was a decent file system for an Arduino device (and to demonstrate the infinite stack recursion problem wouldn't be a gaping security hole problem for them either). I used to think that we would need an IP unencumbered file system, given issues around TomTom and Microsoft, but these days, given how quickly Linux has taken over the embedded and mobile landscape[2] for all but the most tiniest of devices, I don't think that's as important of an issue, since we can just simply use a native linux file system. In the time that it would take to get some other new file system adopted across the industry, it's likely Linux will have enough market share to perhaps compel the other OS vendors to provide interoperability solutions. (Just as the BSD folks have implemented ext2 support; Linux hasn't bothered to implement FFS2 support) - Ted [1] http://www.engadget.com/2012/03/14/samsung-researching-android-based-digital-camera/ [2] http://money.cnn.com/2012/08/08/technology/smartphone-market-share/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
The question is less about the importance of flash drives (or any portable hard drive) today or embedded appliances, but the importance of them in 3-5+ years time (dev time + adoption time). -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
Hi! > > > I also seriously question the niche of people who want to use a thumb > > > drive to transfer > 4GB files. Try it sometime and see what a painful > > > user experience it is > > > > Think for example on consumer devices, for example on most moderns TV > > you can plug a USB memory disk with videos and play them. > > More and more consumer devices, including TV's, are network-enabled. > I'm not at all convinced the USB memory disk model is the one which > makes sense --- you can make a much better user experience work if > you More and more, but USB sticks are still more common... and exchange format would be nice. Currently, I'm just using VFAT + NTFS, but both have disadvantages. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Mon, Aug 20, 2012 at 2:47 AM, Theodore Ts'o wrote: > More and more consumer devices, including TV's, are network-enabled. > I'm not at all convinced the USB memory disk model is the one which > makes sense --- you can make a much better user experience work if you > can rely on networking. That way you don't have to move USB storage > devices around, and USB storage devices are *slow* when the most > common types are HDD's and crappy flash devices. How many people are > going to drop several hundred dollars for a USB-attached SSD, when > using a networking transfer mechanism is much more convenient? My two cents: Flash drives are getting faster as well. Copying an 8GB file to/from a USB drive is not excruciatingly slow and may be quicker and more certain than figuring out how to get a working network connection in some random place, if possible at all. If it is some lousy WiFi with the base station at a distance, a flash drive will be faster. And sometimes people just want to be sure that their data will be at a certain place at a certain time without having to rely on a network that may go down due to external reasons. I also believe LanyFS (assuming that will be well-implemented) would be a nice alternative to exFAT, and of course the kludge that is FAT32. I do have serious doubts about how well it would be adopted. Even if a single major vendor does not want to implement it or worse, actively resists attempts to make it work on their platform, then it already misses its most important goal. Of course, if we give up in advance then it will certainly never happen… Alexander -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sunday 19 August 2012 23:07:32 Raymond Jennings wrote: > On Sun, 2012-08-19 at 20:47 -0400, Theodore Ts'o wrote: > > On Mon, Aug 20, 2012 at 01:06:20AM +0200, Carlos Alberto Lopez Perez wrote: > > > > > > > I also seriously question the niche of people who want to use a thumb > > > > drive to transfer > 4GB files. Try it sometime and see what a painful > > > > user experience it is > > > > > > Think for example on consumer devices, for example on most moderns TV > > > you can plug a USB memory disk with videos and play them. > > > > More and more consumer devices, including TV's, are network-enabled. > > I'm not at all convinced the USB memory disk model is the one which > > makes sense --- you can make a much better user experience work if you > > can rely on networking. That way you don't have to move USB storage > > devices around, and USB storage devices are slow when the most > > common types are HDD's and crappy flash devices. How many people are > > going to drop several hundred dollars for a USB-attached SSD, when > > using a networking transfer mechanism is much more convenient? The cost of such drives will not stay so high and the speed will leap drastically once UAS over USB 3.0 will have become established. And I am sure that you don't want every content to go over the network. And far from everywhere will have the bandwidth to transfer so much data. And of course you assume that you know where you are going to need your content before you leave or have remote access to your base system. You may argue that networked transfers will take up a bigger slice of the cake, but storage devices are far from dead, even for large amounts of data. And the definition of largeness is growing. Regards Oliver -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, 2012-08-19 at 20:47 -0400, Theodore Ts'o wrote: > On Mon, Aug 20, 2012 at 01:06:20AM +0200, Carlos Alberto Lopez Perez wrote: > > > > > I also seriously question the niche of people who want to use a thumb > > > drive to transfer > 4GB files. Try it sometime and see what a painful > > > user experience it is > > > > Think for example on consumer devices, for example on most moderns TV > > you can plug a USB memory disk with videos and play them. > > More and more consumer devices, including TV's, are network-enabled. > I'm not at all convinced the USB memory disk model is the one which > makes sense --- you can make a much better user experience work if you > can rely on networking. That way you don't have to move USB storage > devices around, and USB storage devices are *slow* when the most > common types are HDD's and crappy flash devices. How many people are > going to drop several hundred dollars for a USB-attached SSD, when > using a networking transfer mechanism is much more convenient? > > > And I doubt that the majority of this consumer devices are able to read > > nothing more than FAT32 file-systems, so the 4GB limit is a big problem. > > And here is where Microsoft is pushing their exFAT FS since it allows > > working with 4GB+ files without the NTFS overhead. > > We'll see how popular a heavily IP-encumbered file system will be, > especially given that its main use case is for devices which are so > constrained that they can't afford to use a "real file system" (like > ntfs or ext4 or some other more sophisticated file system), but which > nevertheless needs to be able to handle 4GB+ files. My two cents: After seeing microsoft's attack on TomTom over the vfat patents I honesstly would consider it a good move to have an alternative free format available. > I'm sure there will be some use cases that might fit that niche, but > it seems pretty tiny. And this is completely ignoring what might > happen if in the future people take 1gig fiber connections to the home > (such as what many people in Kansas City will be enjoying very > shortly) for granted > > > As a side note, it would be possible to write a driver for exFAT and get > > it merged upstream on the Linux Kernel without "breaking any law"? > > Goggling I found an attempt to write such driver but seems that never > > got merged: https://lkml.org/lkml/2009/2/8/24 > > You'll need to talk to a lawyer about that, since that's fundamentally > a legal question. > > Regards, > > - Ted > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Mon, Aug 20, 2012 at 01:06:20AM +0200, Carlos Alberto Lopez Perez wrote: > > > I also seriously question the niche of people who want to use a thumb > > drive to transfer > 4GB files. Try it sometime and see what a painful > > user experience it is > > Think for example on consumer devices, for example on most moderns TV > you can plug a USB memory disk with videos and play them. More and more consumer devices, including TV's, are network-enabled. I'm not at all convinced the USB memory disk model is the one which makes sense --- you can make a much better user experience work if you can rely on networking. That way you don't have to move USB storage devices around, and USB storage devices are *slow* when the most common types are HDD's and crappy flash devices. How many people are going to drop several hundred dollars for a USB-attached SSD, when using a networking transfer mechanism is much more convenient? > And I doubt that the majority of this consumer devices are able to read > nothing more than FAT32 file-systems, so the 4GB limit is a big problem. > And here is where Microsoft is pushing their exFAT FS since it allows > working with 4GB+ files without the NTFS overhead. We'll see how popular a heavily IP-encumbered file system will be, especially given that its main use case is for devices which are so constrained that they can't afford to use a "real file system" (like ntfs or ext4 or some other more sophisticated file system), but which nevertheless needs to be able to handle 4GB+ files. I'm sure there will be some use cases that might fit that niche, but it seems pretty tiny. And this is completely ignoring what might happen if in the future people take 1gig fiber connections to the home (such as what many people in Kansas City will be enjoying very shortly) for granted > As a side note, it would be possible to write a driver for exFAT and get > it merged upstream on the Linux Kernel without "breaking any law"? > Goggling I found an attempt to write such driver but seems that never > got merged: https://lkml.org/lkml/2009/2/8/24 You'll need to talk to a lawyer about that, since that's fundamentally a legal question. Regards, - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On 19/08/12 23:04, Theodore Ts'o wrote: >> There is a small niche which LanyFS tries to fit in. It is for those who >> > do not want to bother about how to use a fs when they are in a hurry or >> > when they just want to listen to music in the car. It is for the >> > it-must-be-easy-enough-for-my-gradma fraction. > Music doesn't require > 4GB files, and there are plenty of very easy > to use solutions that utilize streaming over the network. That is > *always* going to be easier than figuring out ahead of time which > files you want, and then manually copying them onto a thumb drive, and > then taking the thumb drive to the car Somehow I can't quite > imagine your grandma manually copying files over using LanyFS. :-) > Microsoft is pushing exFAT [1] as the successor of FAT32 for this kind of use cases. The problem is that exFAT is full of patents and they require you to purchase a license for use. I think that this LanyFS could be a great free alternative to exFAT when the time of 4+GB-for-a-movie will became the norm. The problem will be that IMHO Microsoft won't be interested in implementing this FS on their OS despite of the license since their interest is to push exFAT. > I also seriously question the niche of people who want to use a thumb > drive to transfer > 4GB files. Try it sometime and see what a painful > user experience it is Think for example on consumer devices, for example on most moderns TV you can plug a USB memory disk with videos and play them. And videos are getting bigger and bigger. Many FullHD movies that you can download or record are bigger than 4GB, and in a few years this will be the norm. And I doubt that the majority of this consumer devices are able to read nothing more than FAT32 file-systems, so the 4GB limit is a big problem. And here is where Microsoft is pushing their exFAT FS since it allows working with 4GB+ files without the NTFS overhead. As a side note, it would be possible to write a driver for exFAT and get it merged upstream on the Linux Kernel without "breaking any law"? Goggling I found an attempt to write such driver but seems that never got merged: https://lkml.org/lkml/2009/2/8/24 Regards! [1] http://en.wikipedia.org/wiki/ExFAT signature.asc Description: OpenPGP digital signature
Re: [PATCH] fs: Introducing Lanyard Filesystem
Theodore Ts'o writes: > > In practice, the solution of using either FAT When the answer is FAT the question usually didn't make much sense. -Andi -- a...@linux.intel.com -- Speaking for myself only -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, Aug 19, 2012 at 05:33:52PM +0200, Dan Luedtke wrote: > > Poorly crafted example: > Let's say you have a 6GB video file you want to give somebody (e.g. a > video cutter) on a thumb drive. The cutter wants to edit the file, so he > needs read and write access to it. After cutting the file is to be > played on a TV screen with USB-port. What if the cutter does use two > different, major, non-Linux operating systems, let's say one for cutting > and the other one for adding visual effects? Now imagine the cutter (and > of course the TV screen vendor) don't care much about filesystems as > they are just generating costs (implementation), so it will only come > with minimal compatibility. > - What filesystem would you recommend to share that video file? So the problem with using interoperability as your prime driver is that other operating systems won't have implemented your LanyFS, and because of licensing issues, it will be very hard to get even a shareware distribution of LanyFS for Windows (because of licensing issues around Window's IFS SDK). In practice, the solution of using either FAT for most cases as the interchange format works well enough for most purposes. Granted it doesn't in the case of your example of a 6GB video file (which I'd probably transmit using a networking protocol, since a thumb drive would be really slow.) What I would do if I needed to transfer such a file, and I didn't have access to high speed networking, would be to use ext2, and then either use the ext2 FUSE driver with FUSE for Windows or Macintosh --- or, I would port the userspace e2tools package to the target OS, and use that to access the ext2 file ssytem. And I'd do that because the software is available today, right now, without having to figure out how to port LanyFS to the operating system. > There is a small niche which LanyFS tries to fit in. It is for those who > do not want to bother about how to use a fs when they are in a hurry or > when they just want to listen to music in the car. It is for the > it-must-be-easy-enough-for-my-gradma fraction. Music doesn't require > 4GB files, and there are plenty of very easy to use solutions that utilize streaming over the network. That is *always* going to be easier than figuring out ahead of time which files you want, and then manually copying them onto a thumb drive, and then taking the thumb drive to the car Somehow I can't quite imagine your grandma manually copying files over using LanyFS. :-) I also seriously question the niche of people who want to use a thumb drive to transfer > 4GB files. Try it sometime and see what a painful user experience it is Regards, - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
Il 19/08/2012 18:53, Dan Luedtke ha scritto: On Sun, 2012-08-19 at 15:27 +0100, Al Viro wrote: * unlimited recursion I am already working on that one, but it's tricky. * unlink() does *not* truncate the file contents; I did not know that. I add that vmtruncate is deprecated and I see a call to inode_dio_wait but no support for direct IO. In addition the lock rules seem strange, I would avoid playing with inode->i_lock. * while we are at it, neither of those should free the on-disk inode; again, that should happen only when the inode is evicted. Makes sense now. Thanks! And I think you'll call d_delete two times. As general suggestion: to have a general view how things can work you can look at other fs. Maybe ramfs or tmpfs are simple enough to understand the general concepts. Marco -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, Aug 19, 2012 at 06:53:37PM +0200, Dan Luedtke wrote: > > * minor point, but endianness-flipping in place is *the* way to get > > hard-to-catch endianness bugs. foo = cpu_to_le64(foo) is a bloody bad idea; > > either use object for host-endian all along, or use it only for (in your > > case) little-endian. > I am not sure I understood this right. > At what point should I convert e.g. the file size (little endian 64bit > value stored on disk) to host endianess? When filling the inode? > Is inode->i_size = le64_to_cpu(size) bad, too? Conversions *in* *place* are bad. The above is fine if you have 'size' variable used only for little-endian values (and declare it __le64, then). The thing is, thinking of endianness conversions as architecture-conditional byteswaps is wrong. Treat them as you would treat dealing with some encoding; that's the reason why we have separate le64_to_cpu() and cpu_to_le64(), even though they are identical as functions - on all architectures we support. And don't use the same variable for both the host- and fixed-endian values; it's far too easy to get confused on code modifications and get the situation when two code paths lead to the same point, one with host-endian value in that variable and another with fixed-endian. Real fun to debug, especially when one of the codepaths is rarely taken... -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, 2012-08-19 at 15:27 +0100, Al Viro wrote: > * unlimited recursion I am already working on that one, but it's tricky. > * unlink() does *not* truncate the file contents; I did not know that. > * while we are at it, neither of those should free the on-disk > inode; again, that should happen only when the inode is evicted. Makes sense now. Thanks! > * I might be missing something, but copying a bunch of files > with something like cp /foo/* /mnt seems to be guaranteed to create > really lousy binary tree in target directory (they will go in lexicographical > order and you don't seem to rebalance the tree at all) You missed nothing, there is no rebalancing yet. Thats why performance is bad at the moment as soon as the "tree" stops being a tree. > * you are really abusing iget() there. Noted. Thanks! > * minor point, but endianness-flipping in place is *the* way to get > hard-to-catch endianness bugs. foo = cpu_to_le64(foo) is a bloody bad idea; > either use object for host-endian all along, or use it only for (in your > case) little-endian. I am not sure I understood this right. At what point should I convert e.g. the file size (little endian 64bit value stored on disk) to host endianess? When filling the inode? Is inode->i_size = le64_to_cpu(size) bad, too? Thank you very much for your comments! That'll keep me busy a few weeks. regards, Dan PS: As Jochen Striepe pointed out, lan...@librelist.com behaves badly, I removed it. -- Dan Luedtke http://www.danrl.de -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, Aug 19, 2012 at 05:33:52PM +0200, Dan Luedtke wrote: > Still wondering if anyone bothers to actually look at the code? Some obvious notes: * unlimited recursion is a killer; here its depth is controlled by the fs image contents and it's trivial to cook one that would overflow kernel stack. Seeing that you want to use it for removable media, that's a gaping security hole right there * unlink() does *not* truncate the file contents; file that had been opened and unlinked should keep its contents until it's closed. The same goes for overwriting rename(). * while we are at it, neither of those should free the on-disk inode; again, that should happen only when the inode is evicted. * I might be missing something, but copying a bunch of files with something like cp /foo/* /mnt seems to be guaranteed to create really lousy binary tree in target directory (they will go in lexicographical order and you don't seem to rebalance the tree at all) * you are really abusing iget() there. Leaving the locking issues aside, that's going to get you icache filled with irrelevant stuff. Moreover, it's far too heavy a club; allocating and filling struct inode when all you really need is name and a couple of pointers in the disk block? * minor point, but endianness-flipping in place is *the* way to get hard-to-catch endianness bugs. foo = cpu_to_le64(foo) is a bloody bad idea; either use object for host-endian all along, or use it only for (in your case) little-endian. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
Hi again, On Sun, Aug 19, 2012 at 05:33:52PM +0200, Dan Luedtke wrote: > - What filesystem would you recommend to share that video file? You pointed out you tried ext3, so I thought ext3 was available on your target platforms. I'm sorry I don't understand your reasoning about its shortcomings. So long, Jochen. P.S.: lan...@librelist.com removed from Cc: since it is obviously a closed list, answering my previous email with a subscription confirmation request. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, 2012-08-19 at 15:25 +0200, Marco Stornelli wrote: > Ok, I try to do a summary. You are trying to write a new general and > minimal fs for mobile storage device, minimal enough to be easy ported > on several fs. So at the end you are trying to replace the solution is > used today on many platforms (fat32). Without other consideration about > "no-feature is a feature", it seems to me really challenging because the > project can fail its goal no because there is design problem, bugs and > so on but because of limited use. We are talking about interoperability > problem, and we really know some companies out there from this point of > view :) You nailed it and you made me feel like Don Quijote. I'd like to give it a try anyway. regards, Dan -- Dan Luedtke http://www.danrl.de -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, 2012-08-19 at 14:02 +0200, Jochen Striepe wrote: > You wrote a new fs just because you didn't bother to use the existing > ones as intended? You are over-estimating my motivation. I use many fs as intended, and they do a great job. I would not replace them, not on my workstation and not on my servers. But from time to time I get into trouble when I start transferring data from system A to system B by using removable storage devices. While this can be solved as long as I know what the target platform/os will be, it becomes more difficult when the target platform/os is unknown. Poorly crafted example: Let's say you have a 6GB video file you want to give somebody (e.g. a video cutter) on a thumb drive. The cutter wants to edit the file, so he needs read and write access to it. After cutting the file is to be played on a TV screen with USB-port. What if the cutter does use two different, major, non-Linux operating systems, let's say one for cutting and the other one for adding visual effects? Now imagine the cutter (and of course the TV screen vendor) don't care much about filesystems as they are just generating costs (implementation), so it will only come with minimal compatibility. - What filesystem would you recommend to share that video file? There is a small niche which LanyFS tries to fit in. It is for those who do not want to bother about how to use a fs when they are in a hurry or when they just want to listen to music in the car. It is for the it-must-be-easy-enough-for-my-gradma fraction. It is for those who think that data stored on a thumb drive intended for use with unknown/untrusted/undocumented systems should not be critical data anyway. I do not recommend storing vital files or data worth protecting from unauthorized modification/access on a "highly mobile" thumb drive, especially not one formatted with LanyFS. After all, it is a fs that tries to grant access to the data at any cost*. Still wondering if anyone bothers to actually look at the code? Although I appreciate any feedback about why-would-one-even-write-a-fs, I would also be very happy about comments that help improve the code. regards, Dan *This one is worth a discussion, but LKML might not be the right place? -- Dan Luedtke http://www.danrl.de -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
Il 19/08/2012 15:34, Dan Luedtke ha scritto: On Sun, 2012-08-19 at 12:14 +0200, Marco Stornelli wrote: what are pros and cons of this fs compared with existing fs? Pros: - Simplicity. LanyFS avoids any unnecessary complexity. Example: I had a lot of problems reading and writing data on the Arduino platform. Most of it was, because the platform wasn't capable of dealing with FAT32 when files grew. I don't know if LanyFS performs better on Arduino, I haven't written a driver for Arduino yet, I am glad I managed to get it working on Linux for the moment. - Interoperability. LanyFS was designed to unite those features which are common on most operating systems. [...] All information, including file and directory metadata, is stored in the most purposive format without honoring the habits of any particular operating system. Example: I have a nice FullHD movie file which is about 8GB in size. I wanted to play that video file on a RaspberryPI connected to a TV. FAT32 wasn't able to store the file because it was larger than 4GB. I then tried using ext3, but the ownership information from my workstation (were the file was copied from) did not match the ones the RaspberryPI had, since I usually do not synchronize user profiles between workstations and embedded devices. There are workarounds, one might say, but shouldn't it be much more easier to transfer files from one device to another? There are use cases were modes/permissions and ownership of files does not matter, where quota and accounting isn't necessary, where access times don't need to be stored since they would never get updated anyway. Flexibility. LanyFS adopts to the underlying storage device by adjusting its parameters accordingly. It can address block devices starting from 4 KiB up to 64 ZiB with minimal overhead by parameterizing the filesystem at formatting time. Example: At the university we sometimes work with smart cards and their sometimes "strange" filesystems. I am not an expert on smart cards and stuff like that, but I was being told that a more simple filesystem would be welcome. AFAIK we are talking about "disks" with a size of less than 1MB. I don't think this is a big feature, most other fs can be adjusted to perfectly fit a particular purpose, too. Yet there still is the demand for something "much more simpler". Cons: - LanyFS is featureless. It lacks of features. This is considered a feature by some people, and a big con by others :) - Use of recursion. It may not scale in some cases (e.g. very large files on embedded platforms). Recursion can be a problem, we have got a little stack, I don't know if it's a good idea. - Current restrictions due to early stage. Max blocksize is 4k, but 4MB might be better. The LWN article pointed to by Alan talks about this. Ok, I try to do a summary. You are trying to write a new general and minimal fs for mobile storage device, minimal enough to be easy ported on several fs. So at the end you are trying to replace the solution is used today on many platforms (fat32). Without other consideration about "no-feature is a feature", it seems to me really challenging because the project can fail its goal no because there is design problem, bugs and so on but because of limited use. We are talking about interoperability problem, and we really know some companies out there from this point of view :) Marco -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
Hello, On Sun, Aug 19, 2012 at 03:34:24PM +0200, Dan Luedtke wrote: > tried using ext3, but the ownership information from my workstation > (were the file was copied from) did not match the ones the RaspberryPI > had, since I usually do not synchronize user profiles between > workstations and embedded devices. There are workarounds, one might say, You wrote a new fs just because you didn't bother to use the existing ones as intended? Puzzled, Jochen. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, 2012-08-19 at 12:14 +0200, Marco Stornelli wrote: > You say that you wrote a new fs because of some lacks in the other fs > ("...I kind of invented a very simple filesystem that solves the issues > I had with other filesystems"). I read the website (very quickly > actually) but I didn't find the point: I had to be a little bit stingy with that, I haven't submitted my thesis document yet, and I don't want to publish it before that. I may not be allowed to do so before it is officially submission, IANAL. At the moment, the website lacks of the information you were looking for, sorry for that. The thesis contains also a lot of statistical data and fancy diagrams and stuff like that. I analyzed about 600k file stored on various removable storage devices. 80 volunteers sent in data about their devices, generated by a program (windows) and scripts (linux, bsd, osx) I wrote for that purpose. The data shows that people use more complex filesystems as soon as they are confronted with problems (mostly the 4GB limit). After that they have problems getting their data accessed by other systems. I derived from that, that we need a filesystem that is so simple that even unpopular operating systems can implement it without having their business plan explode. The data will be made public (like open data) after submitting the document. Others are then free to derive statements from it, may then verify or contradict my conclusions. > what are pros and cons of this fs > compared with existing fs? Pros: - Simplicity. LanyFS avoids any unnecessary complexity. Example: I had a lot of problems reading and writing data on the Arduino platform. Most of it was, because the platform wasn't capable of dealing with FAT32 when files grew. I don't know if LanyFS performs better on Arduino, I haven't written a driver for Arduino yet, I am glad I managed to get it working on Linux for the moment. - Interoperability. LanyFS was designed to unite those features which are common on most operating systems. [...] All information, including file and directory metadata, is stored in the most purposive format without honoring the habits of any particular operating system. Example: I have a nice FullHD movie file which is about 8GB in size. I wanted to play that video file on a RaspberryPI connected to a TV. FAT32 wasn't able to store the file because it was larger than 4GB. I then tried using ext3, but the ownership information from my workstation (were the file was copied from) did not match the ones the RaspberryPI had, since I usually do not synchronize user profiles between workstations and embedded devices. There are workarounds, one might say, but shouldn't it be much more easier to transfer files from one device to another? There are use cases were modes/permissions and ownership of files does not matter, where quota and accounting isn't necessary, where access times don't need to be stored since they would never get updated anyway. Flexibility. LanyFS adopts to the underlying storage device by adjusting its parameters accordingly. It can address block devices starting from 4 KiB up to 64 ZiB with minimal overhead by parameterizing the filesystem at formatting time. Example: At the university we sometimes work with smart cards and their sometimes "strange" filesystems. I am not an expert on smart cards and stuff like that, but I was being told that a more simple filesystem would be welcome. AFAIK we are talking about "disks" with a size of less than 1MB. I don't think this is a big feature, most other fs can be adjusted to perfectly fit a particular purpose, too. Yet there still is the demand for something "much more simpler". Cons: - LanyFS is featureless. It lacks of features. This is considered a feature by some people, and a big con by others :) - Use of recursion. It may not scale in some cases (e.g. very large files on embedded platforms). - Current restrictions due to early stage. Max blocksize is 4k, but 4MB might be better. The LWN article pointed to by Alan talks about this. - No use of MTD/UBI features. LanyFS expects a block layer abstraction beneath it, this is because it targets highly mobile devices, and those devices also happen to be rotating disk (external hard drives) sometimes. It is not optimized for flash-only devices, although most of the devices it may run on one day will probably be flash-based. > What problems does it solve? - I can now watch >4GB movie files from USB thumb drive :) - Providing a storage device interface will be easier for vendors. TVs, car entertainment systems, digital cameras, sensor devices, navigation systems and all kind of other devices often use FAT32 since it just works on everyones computer, but it is limit in size. Other fs are there to hop in, but vendors often refuse to implement them since they have to be compatible with a lot of features. They won't face such problems with LanyFS, it just has no features. Of course, I am just a student, I have no money and
Re: [PATCH] fs: Introducing Lanyard Filesystem
Il 19/08/2012 12:12, Dan Luedtke ha scritto: (resent) On Sun, 2012-08-19 at 00:16 +0200, richard -rw- weinberger wrote: What are the use cases of this filesystem? It looks very minimal without much features. That's the feature, actually. Think of the Arduino platform or other embedded devices (TV, Car Entertainment) that just want to read/play files. You're right, no big features, but that's by design. Dan You say that you wrote a new fs because of some lacks in the other fs ("...I kind of invented a very simple filesystem that solves the issues I had with other filesystems"). I read the website (very quickly actually) but I didn't find the point: what are pros and cons of this fs compared with existing fs? What problems does it solve? Marco -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
(resent) On Sun, 2012-08-19 at 00:16 +0200, richard -rw- weinberger wrote: > What are the use cases of this filesystem? > It looks very minimal without much features. That's the feature, actually. Think of the Arduino platform or other embedded devices (TV, Car Entertainment) that just want to read/play files. You're right, no big features, but that's by design. Dan -- Sent from my mobile phone. Please excuse my brevity. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, Aug 19, 2012 at 12:22 AM, Dan Luedtke wrote: > That's the feature, actually. Think of the Arduino platform or other > embedded devices (TV, Car Entertainment) that just want to read/play files. > You're right, no big features, but that's by design. So your fs is designed to run on top of NAND/NOR flash memory? -- Thanks, //richard -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, Aug 19, 2012 at 1:38 AM, Dan Luedtke wrote: > This patch introduces the Lanyard Filesystem (LanyFS), a filesystem > for highly mobile and removable storage devices. What are the use cases of this filesystem? It looks very minimal without much features. -- Thanks, //richard -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH] fs: Introducing Lanyard Filesystem
On Sun, 19 Aug 2012 01:38:37 +0200 Dan Luedtke wrote: > This patch introduces the Lanyard Filesystem (LanyFS), a filesystem > for highly mobile and removable storage devices. > > Signed-off-by: Dan Luedtke > --- > "Release early, release often" they said. Here is my work of the > past weeks. This is a RFC patch, so please comment, criticize, suggest > and test test test. Of course patches are welcome, too. Starting from first principles a bit you might want to read/review Arnd Bergmanns work on Flash drive performance and behaviour. It has some fairly significant impact on how you lay out your file system and in particular questions like "where should I put stuff that is updated regularly' (in the same blocks a FAT fs would put the FAT being one clear answer) http://lwn.net/Articles/428584/ and a pile of other stuff. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/