Re: [Freedos-devel] compressed FAT filesystems
Hi Imre, > Which free/open compressed embedded filesystems would that be? Browsing wikipedia a bit gives me quite a few READONLY free and open source compressed filesystems, several of which are useful for embedded systems. For writeability, you would typically have to alloc new space in your filesystem whenever a changed cluster no longer fits (in compressed form) into where it was before. Then you need the ability to have the storage locations of the cluster contents be nonlinear, and you need some way to find unused areas and either re-use them on the fly or defrag them out of the filesystem with an "offline" (while the filesystem is not used) tool. Luckily a readonly compressed filesystem is very useful for various things already, so you can make people happy with one of the simple readonly systems listed below :-). http://en.wikipedia.org/wiki/Cramfs - for ROM, GPLed, uses zlib, each "page" is compressed separately, metadata is not compressed, max 16 MB / file, 256 MB / filesystem http://en.wikipedia.org/wiki/SquashFS - GPLed, readonly, uses gzip, used for several LiveCD versions of Linuxes, compresses data and metadata in chunks of up to 1 MB each... Max 2^64 bytes per file. Default block size 128 kB. http://en.wikipedia.org/wiki/Zisofs - Linux, basically ISO9660 with extra flag to mark compressed files, it seems ;-). This means that uncompressed files on Zisofs can be accessed by any ISO9660 capable driver and operating system :-). http://en.wikipedia.org/wiki/Cloop - Linux, simply a wrapper to store a compressed block device image on a block device, it seems. Compressed images contain a "seek index with compressed and un- compressed block sizes in pairs" and the zlib compressed blocks. I assume you get the offset of a compressed block by summing up the sizes of the compressed blocks before it. Wikipedia says that Apple DMG has a similar design. http://logfs.org/logfs/LogFS - GPLed successor for JFFS2, for flash devices... Seems to be "work in progress" http://lxr.linux.no/linux/Documentation/filesystems/romfs.txt A really simple filesystem: Main content is a linked list of files, each file consisting of a header and the data of the file, all aligned to 16 byte boundaries. Header contains: Link, flags, size, checksum, max 15 bytes of name. You can easily introduce a flag for compressed files. The link has 28 bit (high 28 bit of offset), leaving 36 bits for flags. Flags are type (hardlink, dir, file, symlink, block device, char device, socket, fifo) plus extra info. Hardlink info: Offset pointer. Dir info: First file pointer. Device info: The major and minor number. Symlink info: None - instead, the "file content" is the symlink string. The "." and ".." of directories are stored as normal directory entries, the same idea as in FAT. > Add a compression attribute to the inode structure in LEAN. And then > put the compressed data into the file. Tools have done the same for FAT: Use an unused bit or combination of bits in the file attribute to mark a file as compressed, or use some sort of header in the file contents... The suggestion of my previous mail is a bit similar to CramFS and Cloop in design, but my main idea is to have a filesystem which stays similar to FAT and where it is easy to transform a FAT system into a compressed one with a tool and for the other way round make a compressed FAT system look like it was a normal one with a driver for the other way round, making it possible for the FreeDOS kernel to "mount" it easily :-). This includes keeping metadata very FATish, adding compression related metadata at a low profile place (eg replace the 2nd copy of the FAT) and compress only the data clusters. Eric :-) - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods (OT)
Hi Imre, nice explanation, thanks :-). > Here are the numbers for the data structures used by defrag, > when calculated for FAT32. You mean for a worst case FAT32 filesystem with 2^28 (256M) clusters where the FAT on disk is also 1 GB per copy... > Cluster movable map: 32 MB (fat size * 1 bit) > FAT refered map: 32MB (fat size * 1 bit) > FAT refered table: 1024MB (fat size * 4 byte) > Dir refered table (take 10 files): > (# files * 6 bytes) 0,5 MB > total: 1088,5 MB (and I though XMS only went to 64MB) So its dominated by "one FAT32 copy plus small stuff" :-) Actually XMS2 goes to 64 MB, but on a 286, you can only have 16 MB. When you use XMS3, you get up to 4 GB, but the interface of XMS3 uses 32bit registers so if your software uses XMS3, it no longer runs on 286 or older ;-). > Cluster movable map: indicates which clusters are movable. Okay :-). Are many clusters non movable? > FAT refered map: indicates which clusters are refered to in the FAT What does that mean? > FAT refered table: stores the back pointers for the FAT clusters Interesting. You could probably make a sparse variant of this by making the default assumption "previous cluster or none points to this cluster" and only storing the exceptions to that rule in a hashtable. Whether it is previous or rather none is easily found out by looking at the FAT. > DIR refered table: stores the back pointers for the directory > refered clusters Useful and quite small :-) Eric - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Bulgarian keyboard driver for Blocek
I am sorry - my fault. Problem was caused by cause sensitive filenames on server. It is fixed now. Try it again, please. - Original Message - From: "Jim Hall" <[EMAIL PROTECTED]> To: Sent: Tuesday, March 18, 2008 9:44 PM Subject: Re: [Freedos-devel] Bulgarian keyboard driver for Blocek > On 3/18/08, Ladislav Lacina <[EMAIL PROTECTED]> wrote: > > Dimitar Mitov kindly provided the bulgarian keyboard driver for text editor > > Blocek. The archives and webpage are updated. > > http://www.laaca-mirror.ic.cz > > > > I mirrored this to the ibiblio archive. However, I noticed that the > binary rar file hasn't changed from the previous 1.33b. Only the > source rar file is different in this new "1.33b". Is that right? I > think you forgot to update the binary rar archive. > > $ ls -l blocek133b* source/blocek133b* > -rw-r--r-- 1 freedos users 298505 Mar 18 16:38 blocek133b2.rar > -rw-r--r-- 1 freedos users 298505 Nov 3 15:19 blocek133b.rar > -rw-r--r-- 1 freedos users 1545674 Mar 18 16:39 source/blocek133b2_s.rar > -rw-r--r-- 1 freedos users 190797 Nov 8 19:22 source/blocek133b_s.rar > > $ md5sum blocek133b* source/blocek133b* > d5f72fce7dc546453c37a87b86337728 blocek133b2.rar > d5f72fce7dc546453c37a87b86337728 blocek133b.rar > ad0b612d35343cdd900bd214d28ef93d source/blocek133b2_s.rar > f8e1df6ffcdd6a398a5534ee953458e7 source/blocek133b_s.rar > > > Since you've basically re-released 1.33b, when I mirrored this to > ibiblio I used the version code "133b2" (compared to "133b") to > indicate that this was a slight update. > > > -jh > > - > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ > ___ > Freedos-devel mailing list > Freedos-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/freedos-devel - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods (OT)
Take the following cluster chain: 2 -> 3 -> 4 -> EOF Then the FAT refered table contains: table[4] -> 3 table[3] -> 2 table[2] -> 0 Well on FAT32 the labels are normaly 28bit, but that shouldn't realy matter. The FAT refered map is used to directly know wether to consult the FAT refered table or the DIR refered table. Imre >- Oorspronkelijk bericht - >Van: Alain M. [mailto:[EMAIL PROTECTED] >Verzonden: woensdag, maart 19, 2008 07:49 PM >Aan: freedos-devel@lists.sourceforge.net >Onderwerp: Re: [Freedos-devel] Fwd: freedos defrag methods (OT) > >Interesting... > >What information is tored in the "FAT refered table" it needs 32 bits? > >Alain > >Imre Leber escreveu: >> Here are the numbers for the data structures used by defrag, when calculated >> for FAT32. >> >> Cluster movable map: 32 MB (fat size * 1 bit) >> FAT refered map: 32MB (fat size * 1 bit) >> FAT refered table: 1024MB (fat size * 4 byte) >> Dir refered table (take 10 files): (# files * 6 bytes) 0,5 MB >> >> total: 1088,5 MB (and I though XMS only went to 64MB) >> >> In depth: >> >> Cluster movable map: indicates which clusters are movable. >> FAT refered map: indicates which clusters are refered to in the FAT >> FAT refered table: stores the back pointers for the FAT clusters >> DIR refered table: stores the back pointers for the directory refered >> clusters >> >> Imre >> >>> - Oorspronkelijk bericht - >>> Van: Eric Auer [mailto:[EMAIL PROTECTED] >>> Verzonden: dinsdag, maart 18, 2008 10:22 PM >>> Aan: freedos-devel@lists.sourceforge.net >>> Onderwerp: Re: [Freedos-devel] Fwd: freedos defrag methods (OT) >>> >>> >>> Hi! >>> For this it is very sad that FreeDOS does not use the LEAN file system (http://freedos-32.sourceforge.net/doc/lean.html) instead. That has back pointers for pretty much everything and would as a consequence be much faster. >>> A lean ad ;-). But you can just as well build data structures with the >>> back pointers in RAM while defrag and similar tools are running. Takes >>> roughly the same amount of RAM as the metadata (FAT, dir) takes on >>> disk. As you often copy those to RAM for work like defragging to get >>> fast access, you basically get some overhead for making the pointers >>> plus you double the amount of RAM that defrag consumes ;-). >>> >>> Note that for example DOSFSCK takes several times the size of your FAT >>> plus several times the size of all directory entries of RAM, too. >>> Actually this would be interesting to know more about: Could some of >>> you run DOSFSCK 2.11c with the -v option on large filesystems, with >>> many files and/or many clusters, and report how much heap it tells >>> you that it has used? It is displayed when DOSFSCK is done. Seems to >>> be 5..10 times the FAT size for systems with few files here... Some >>> "bad case of FAT16" would be interesting (might use 1-5 MB maybe?) >>> as well as, of course, typical FAT32 filesystems. You can find >>> DOSFSCK version 2.11c on my page, as usual: >>> www.coli.uni-saarland.de/~eric/stuff/soft/by-others/dosdosfsck-2.11c.zip >>> (you should already have cwsdpmi anyway, otherwise see e.g. my page) >>> >>> Eric >>> > >- >This SF.net email is sponsored by: Microsoft >Defy all challenges. Microsoft(R) Visual Studio 2008. >http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ >___ >Freedos-devel mailing list >Freedos-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/freedos-devel > > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods (OT)
Interesting... What information is tored in the "FAT refered table" it needs 32 bits? Alain Imre Leber escreveu: > Here are the numbers for the data structures used by defrag, when calculated > for FAT32. > > Cluster movable map: 32 MB (fat size * 1 bit) > FAT refered map: 32MB (fat size * 1 bit) > FAT refered table: 1024MB (fat size * 4 byte) > Dir refered table (take 10 files): (# files * 6 bytes) 0,5 MB > > total: 1088,5 MB (and I though XMS only went to 64MB) > > In depth: > > Cluster movable map: indicates which clusters are movable. > FAT refered map: indicates which clusters are refered to in the FAT > FAT refered table: stores the back pointers for the FAT clusters > DIR refered table: stores the back pointers for the directory refered clusters > > Imre > >> - Oorspronkelijk bericht - >> Van: Eric Auer [mailto:[EMAIL PROTECTED] >> Verzonden: dinsdag, maart 18, 2008 10:22 PM >> Aan: freedos-devel@lists.sourceforge.net >> Onderwerp: Re: [Freedos-devel] Fwd: freedos defrag methods (OT) >> >> >> Hi! >> >>> For this it is very sad that FreeDOS does not use the LEAN file system >>> (http://freedos-32.sourceforge.net/doc/lean.html) instead. That has back >>> pointers for pretty much everything and would as a consequence be much >>> faster. >> A lean ad ;-). But you can just as well build data structures with the >> back pointers in RAM while defrag and similar tools are running. Takes >> roughly the same amount of RAM as the metadata (FAT, dir) takes on >> disk. As you often copy those to RAM for work like defragging to get >> fast access, you basically get some overhead for making the pointers >> plus you double the amount of RAM that defrag consumes ;-). >> >> Note that for example DOSFSCK takes several times the size of your FAT >> plus several times the size of all directory entries of RAM, too. >> Actually this would be interesting to know more about: Could some of >> you run DOSFSCK 2.11c with the -v option on large filesystems, with >> many files and/or many clusters, and report how much heap it tells >> you that it has used? It is displayed when DOSFSCK is done. Seems to >> be 5..10 times the FAT size for systems with few files here... Some >> "bad case of FAT16" would be interesting (might use 1-5 MB maybe?) >> as well as, of course, typical FAT32 filesystems. You can find >> DOSFSCK version 2.11c on my page, as usual: >> www.coli.uni-saarland.de/~eric/stuff/soft/by-others/dosdosfsck-2.11c.zip >> (you should already have cwsdpmi anyway, otherwise see e.g. my page) >> >> Eric >> - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods (OT)
Here are the numbers for the data structures used by defrag, when calculated for FAT32. Cluster movable map: 32 MB (fat size * 1 bit) FAT refered map: 32MB (fat size * 1 bit) FAT refered table: 1024MB (fat size * 4 byte) Dir refered table (take 10 files): (# files * 6 bytes) 0,5 MB total: 1088,5 MB (and I though XMS only went to 64MB) In depth: Cluster movable map: indicates which clusters are movable. FAT refered map: indicates which clusters are refered to in the FAT FAT refered table: stores the back pointers for the FAT clusters DIR refered table: stores the back pointers for the directory refered clusters Imre >- Oorspronkelijk bericht - >Van: Eric Auer [mailto:[EMAIL PROTECTED] >Verzonden: dinsdag, maart 18, 2008 10:22 PM >Aan: freedos-devel@lists.sourceforge.net >Onderwerp: Re: [Freedos-devel] Fwd: freedos defrag methods (OT) > > >Hi! > >> For this it is very sad that FreeDOS does not use the LEAN file system >> (http://freedos-32.sourceforge.net/doc/lean.html) instead. That has back >> pointers for pretty much everything and would as a consequence be much >> faster. > >A lean ad ;-). But you can just as well build data structures with the >back pointers in RAM while defrag and similar tools are running. Takes >roughly the same amount of RAM as the metadata (FAT, dir) takes on >disk. As you often copy those to RAM for work like defragging to get >fast access, you basically get some overhead for making the pointers >plus you double the amount of RAM that defrag consumes ;-). > >Note that for example DOSFSCK takes several times the size of your FAT >plus several times the size of all directory entries of RAM, too. >Actually this would be interesting to know more about: Could some of >you run DOSFSCK 2.11c with the -v option on large filesystems, with >many files and/or many clusters, and report how much heap it tells >you that it has used? It is displayed when DOSFSCK is done. Seems to >be 5..10 times the FAT size for systems with few files here... Some >"bad case of FAT16" would be interesting (might use 1-5 MB maybe?) >as well as, of course, typical FAT32 filesystems. You can find >DOSFSCK version 2.11c on my page, as usual: >www.coli.uni-saarland.de/~eric/stuff/soft/by-others/dosdosfsck-2.11c.zip >(you should already have cwsdpmi anyway, otherwise see e.g. my page) > >Eric > > > >- >This SF.net email is sponsored by: Microsoft >Defy all challenges. Microsoft(R) Visual Studio 2008. >http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ >___ >Freedos-devel mailing list >Freedos-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/freedos-devel > > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods (OT)
>- Oorspronkelijk bericht - >Van: Eric Auer [mailto:[EMAIL PROTECTED] >Verzonden: dinsdag, maart 18, 2008 10:22 PM >Aan: freedos-devel@lists.sourceforge.net >Onderwerp: Re: [Freedos-devel] Fwd: freedos defrag methods (OT) > > >Hi! > >> For this it is very sad that FreeDOS does not use the LEAN file system >> (http://freedos-32.sourceforge.net/doc/lean.html) instead. That has back >> pointers for pretty much everything and would as a consequence be much >> faster. > >A lean ad ;-). LEAN was designed from the ground up for the FreeDOS project?! Imre >Eric > > > >- >This SF.net email is sponsored by: Microsoft >Defy all challenges. Microsoft(R) Visual Studio 2008. >http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ >___ >Freedos-devel mailing list >Freedos-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/freedos-devel > > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] compressed FAT filesystems (was: defrag methods)
>- Oorspronkelijk bericht - >Van: Aitor Santamar?a [mailto:[EMAIL PROTECTED] >Verzonden: woensdag, maart 19, 2008 01:45 AM >Aan: freedos-devel@lists.sourceforge.net >Onderwerp: Re: [Freedos-devel] compressed FAT filesystems (was: defrag methods) > >One more thing: the big deal about DoubleSpace/DriveSpace is not the >compression itself, which could of course increase your free space, >but mostly comes from the variable-sector-per-cluster stuff: you get >lots of extra free space by cutting the space occupied by the many, >say, less than 1KB text files in a 8KB/cluster FAT fs. > Yes, and LEAN doesn't use clusters. So the maximum you can loose per file is 511 bytes, not 8KB. Imre >Just my 2c... >AItor > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] compressed FAT filesystems (was: defrag methods)
>- Oorspronkelijk bericht - >Van: Eric Auer [mailto:[EMAIL PROTECTED] >Verzonden: dinsdag, maart 18, 2008 10:38 PM >Aan: freedos-devel@lists.sourceforge.net >Onderwerp: [Freedos-devel] compressed FAT filesystems (was: defrag methods) > > >Hi! > >> Maybe with adding compression it could make a replacement for >> doublespace? (running it through the network redirector obviously) > >Nope, doublespace compresses the actual data on your disk, >not the space which is free anyway ;-). I think it would be >nice to have a driver for one of those free / open compressed >embedded filesystems for DOS. Some ad-hoc suggestions for a >new filesystem. Which free/open compressed embedded filesystems would that be? Serieus, the other day I read an entire thread on an electronics forum about there not being any of those. (That is why I have completed my own file system, FAT12/FAT16/FAT32 with LFN support, with C interface). Further more that is also what I intended to say. Add a compression attribute to the inode structure in LEAN. And then put the compressed data into the file. Further more, the only reason one would use FAT is that it is compatible with a 1001 number of tools. If you are going to make an incompatible system, at least base it on a more sensible design. That way you also don't have to care about microsoft patents and stuff. Imre >Eric > > > >- >This SF.net email is sponsored by: Microsoft >Defy all challenges. Microsoft(R) Visual Studio 2008. >http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ >___ >Freedos-devel mailing list >Freedos-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/freedos-devel > > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
