Re: [Server-devel] ext2 vs ext4 vs exFAT for XO content SD cards?
Thanks You Arnd, we have many months work ahead of us and this will be a big help! On Thu, Aug 20, 2015 at 4:55 PM, Arnd Bergmann wrote: > On Saturday 15 August 2015 21:05:48 Adam Holt wrote: > > What filesystem would people recommend for ~128GB SD cards inserted into > XO > > laptops or XSCE servers heading far afield, to insert very large > > (evolving?) digital libraries just like Internet-in-a-Box? > > > > Multiyear reliability for all this content would be important, but some > say > > ext4 (slow journaled filesystems?) have their own problems on SD? > > ext4 is better than most file systems in this regard (much better than > ext3, > also better than btrfs). > > > Is wear-leveling of modern SD cards (Flash memory) fully taken care of by > > the largest manufacturers/drivers already, or should we seek out > particular > > filesystems/drivers? And unmount / power off carefully etc? > > Some cards handle it better than others. Traditionally, Kingston cards > were completely unusable after writing data on the order of the card size, > while Samsung and Sandisk did much better (I have not checked in a few > years > though, so YMMV). Most other manufacturers have multiple sources for the > embedded controllers on the cards, so it's a bit of hit and miss. The > controllers have all evolved a lot in the past few years, the simple > controllers that used to be on 8GB cards can no longer cope with the > increased block sizes on today's cards (32GB and larger). > > > In-country copying and eventually in-country remixing of SD cards > > (containing local-language, local-vid cultural jewels) would be > Absolutely > > Wonderful -- if we can achieve that by using a filesystem that works > > instantly (exFAT? NTFS?) across all Windows and Mac computers ideally? > For > > all static content anyway, on 1 partition of several? Accelerating > dup'ing > > of SD cards as a bonus?! > > > > Many paths possible for sure, but most important: what > > interop/longevity/other issues are we overlooking?? > > SDXC cards are mandated to be using exFAT (just like SDHC cards have > to use VFAT, and indeed this is the only difference between the two) > by the SD card standard. If you don't use this, you are strictly speaking > in violation of the standard and the cards might not behave as designed. > In particular, the card is allowed to only do efficient garbage collection > for the access patterns that you get with a single exFAT partition that > spans the entire card and has all its metadata aligned exactly in the > way that the spec defines. > In practice, things tend to work mostly ok with other file systems, but > if you use NTFS or ext3 (rather than ext4), you are usually asking for > trouble. > > The best longevity would be provided by f2fs, which is designed to work > fine on most SD cards. The downside is that it only works on relatively > modern Linux kernels (3.x or higher). > > I would expect that cards today use only dynamic wear leveling, not > static wear leveling as real SSDs do. This means that content on a > read-only partition will decay with the normal life of the card > (several years, but depending on the quality of the card and the > environmental conditions, e.g. not too hot), independent of the > presence of partitions you write to. > > Dynamic wear leveling works best if you have a lot of unused blocks, > so a good strategy for long life would be to leave a whole partition > (e.g. 20% of the size of the writable partition, the more you have, > the longer the card will survive) that never gets written after > manufacturing, or at least gets erased using the fitrim ioctl > command after the initial imaging. > > For a 128 GB card with 115GB of actual space, you could then use something > like: > > * 80GB zisofs/cramfs/squashfs for static data > * 30GB f2fs/ext4 for writable data > * 5GB unused space for wear leveling > > Arnd ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] ext2 vs ext4 vs exFAT for XO content SD cards?
No, it won't help. 1. reading data an SD card (or eMMC, or USB flash drives) does cause writes internal to the card, and does reduce life, 2. there's no such flag to set in an MBR partition table, Perhaps you mean a write protect switch on the card? This is a plastic slide, sensed by a switch in the socket, and provides _advice_ to the system. Some systems don't receive the advice. The system can ignore that advice. Even if acted upon, it only stops change to the data, it doesn't stop writes internal to the firmware of the card. On Sun, Aug 16, 2015 at 05:18:01PM -0700, Sameer Verma wrote: > James, > Would it help to mark the content partition(s) as read only? > > Sameer > > On Aug 16, 2015 5:13 PM, "James Cameron" <[1]qu...@laptop.org> wrote: > > Thanks, interesting questions. > > No, ext4 is not a slow journaled filesystem, and no, there are no > obvious problems on SD when using ext4 given your use case. But it > isn't operating system portable, and as your content is static no need > for a journal. Other features of ext4 make mounting or filesystem > check faster. > > Yes, wear-leveling is taken care of by the firmware in the card, put > there by the manufacturers. Wear-levelling also critical during > reading, since a flash page can't be read repeatedly without > disturbance eventually requiring it to be written to a freshly erased > page. This is all handled by the firmware. Happens way more > frequently than it does on a hard drive. > > Duplication time of SD cards won't be affected by your filesystem or > partition decision. > > One partition is sufficient. MBR partition type best, for > compatibility across the operating systems. > > For filesystem, use FAT32, mounted read-only. FAT32 works across most > Windows and Mac computers, at media sizes up to 2 TB, for file sizes > up to 4 GB. > > Where content cannot live on FAT32 due to file name character set or > metadata, it can be placed in disk image bundles of ISO-9660, > squashfs, or ext4 and loop mounted. The content curation process for > the end user might be easier if bundles can be added and removed as > needed. > > What you might be overlooking; I/O bandwidth of the connection to the > media, endurance impact of reading data from the card slowing > performance one year on, backups, content bundle tamper checks, risk > of filesystem format incompatibilities introduced by new versions of > operating systems after your product is in the field, risk of cross > system malware infections, electrostatic discharge damage to the card, > and how modern cards can change performance behaviour as a result of a > production state awareness flag stored by card firmware. > > On the other hand the alternatives have their own problems. > > -- > James Cameron > [2]http://quozl.linux.org.au/ > ___ > Server-devel mailing list > [3]Server-devel@lists.laptop.org > [4]http://lists.laptop.org/listinfo/server-devel > > References: > > [1] mailto:qu...@laptop.org > [2] http://quozl.linux.org.au/ > [3] mailto:Server-devel@lists.laptop.org > [4] http://lists.laptop.org/listinfo/server-devel > ___ > Devel mailing list > de...@lists.laptop.org > http://lists.laptop.org/listinfo/devel -- James Cameron http://quozl.linux.org.au/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] ext2 vs ext4 vs exFAT for XO content SD cards?
James, Would it help to mark the content partition(s) as read only? Sameer On Aug 16, 2015 5:13 PM, "James Cameron" wrote: > Thanks, interesting questions. > > No, ext4 is not a slow journaled filesystem, and no, there are no > obvious problems on SD when using ext4 given your use case. But it > isn't operating system portable, and as your content is static no need > for a journal. Other features of ext4 make mounting or filesystem > check faster. > > Yes, wear-leveling is taken care of by the firmware in the card, put > there by the manufacturers. Wear-levelling also critical during > reading, since a flash page can't be read repeatedly without > disturbance eventually requiring it to be written to a freshly erased > page. This is all handled by the firmware. Happens way more > frequently than it does on a hard drive. > > Duplication time of SD cards won't be affected by your filesystem or > partition decision. > > One partition is sufficient. MBR partition type best, for > compatibility across the operating systems. > > For filesystem, use FAT32, mounted read-only. FAT32 works across most > Windows and Mac computers, at media sizes up to 2 TB, for file sizes > up to 4 GB. > > Where content cannot live on FAT32 due to file name character set or > metadata, it can be placed in disk image bundles of ISO-9660, > squashfs, or ext4 and loop mounted. The content curation process for > the end user might be easier if bundles can be added and removed as > needed. > > What you might be overlooking; I/O bandwidth of the connection to the > media, endurance impact of reading data from the card slowing > performance one year on, backups, content bundle tamper checks, risk > of filesystem format incompatibilities introduced by new versions of > operating systems after your product is in the field, risk of cross > system malware infections, electrostatic discharge damage to the card, > and how modern cards can change performance behaviour as a result of a > production state awareness flag stored by card firmware. > > On the other hand the alternatives have their own problems. > > -- > James Cameron > http://quozl.linux.org.au/ > ___ > Server-devel mailing list > Server-devel@lists.laptop.org > http://lists.laptop.org/listinfo/server-devel > ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
Re: [Server-devel] ext2 vs ext4 vs exFAT for XO content SD cards?
Thanks, interesting questions. No, ext4 is not a slow journaled filesystem, and no, there are no obvious problems on SD when using ext4 given your use case. But it isn't operating system portable, and as your content is static no need for a journal. Other features of ext4 make mounting or filesystem check faster. Yes, wear-leveling is taken care of by the firmware in the card, put there by the manufacturers. Wear-levelling also critical during reading, since a flash page can't be read repeatedly without disturbance eventually requiring it to be written to a freshly erased page. This is all handled by the firmware. Happens way more frequently than it does on a hard drive. Duplication time of SD cards won't be affected by your filesystem or partition decision. One partition is sufficient. MBR partition type best, for compatibility across the operating systems. For filesystem, use FAT32, mounted read-only. FAT32 works across most Windows and Mac computers, at media sizes up to 2 TB, for file sizes up to 4 GB. Where content cannot live on FAT32 due to file name character set or metadata, it can be placed in disk image bundles of ISO-9660, squashfs, or ext4 and loop mounted. The content curation process for the end user might be easier if bundles can be added and removed as needed. What you might be overlooking; I/O bandwidth of the connection to the media, endurance impact of reading data from the card slowing performance one year on, backups, content bundle tamper checks, risk of filesystem format incompatibilities introduced by new versions of operating systems after your product is in the field, risk of cross system malware infections, electrostatic discharge damage to the card, and how modern cards can change performance behaviour as a result of a production state awareness flag stored by card firmware. On the other hand the alternatives have their own problems. -- James Cameron http://quozl.linux.org.au/ ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
[Server-devel] ext2 vs ext4 vs exFAT for XO content SD cards?
What filesystem would people recommend for ~128GB SD cards inserted into XO laptops or XSCE servers heading far afield, to insert very large (evolving?) digital libraries just like Internet-in-a-Box? Multiyear reliability for all this content would be important, but some say ext4 (slow journaled filesystems?) have their own problems on SD? Is wear-leveling of modern SD cards (Flash memory) fully taken care of by the largest manufacturers/drivers already, or should we seek out particular filesystems/drivers? And unmount / power off carefully etc? In-country copying and eventually in-country remixing of SD cards (containing local-language, local-vid cultural jewels) would be Absolutely Wonderful -- if we can achieve that by using a filesystem that works instantly (exFAT? NTFS?) across all Windows and Mac computers ideally? For all static content anyway, on 1 partition of several? Accelerating dup'ing of SD cards as a bonus?! Many paths possible for sure, but most important: what interop/longevity/other issues are we overlooking?? ___ Server-devel mailing list Server-devel@lists.laptop.org http://lists.laptop.org/listinfo/server-devel
