Re: [Freedos-devel] compressed FAT filesystems (was: defrag methods)
One more thing: the big deal about DoubleSpace/DriveSpace is not the compression itself, which could of course increase your free space, but mostly comes from the variable-sector-per-cluster stuff: you get lots of extra free space by cutting the space occupied by the many, say, less than 1KB text files in a 8KB/cluster FAT fs. Just my 2c... AItor 2008/3/18, Eric Auer <[EMAIL PROTECTED]>: > > Hi! > > > Maybe with adding compression it could make a replacement for > > doublespace? (running it through the network redirector obviously) > > Nope, doublespace compresses the actual data on your disk, > not the space which is free anyway ;-). I think it would be > nice to have a driver for one of those free / open compressed > embedded filesystems for DOS. Some ad-hoc suggestions for a > new filesystem: > > Keep a normal FAT and boot sector and stuff, but wrap all the > access to the data clusters. For each data cluster, keep some > pointer to where it ACTUALLY starts (granularity: cluster, sector > or byte) and allow the clusters to be compressed. You would need > some offline "make compressed version of this filesystem" tool > and you would not be able to write to the filesystem later. > To get a relatively simple implementation, you can assume that > each cluster is separately compressed, for example with LZSS. > Then you only look at one cluster at a time for uncompressing. > > Max amount of extra metadata: 32 bits per cluster. Note that > you do not store the size of the compressed cluster nor any > "this cluster is not compressed flag" - you can easily derive > those from the difference between this and next cluster start. > > To reduce the space taken by the extra metadata, you can use > a more "granular" offset for the actual data location, and > you can compress the offset array by only storing the size of > each compressed cluster instead of its offset. To avoid too > much speed loss, you could still keep the offset of every Nth > cluster in some array in RAM to reduce the needed "counting". > You can even store the size of each compressed cluster in a > "granular" way, eg by saying that each compressed cluster has > to start at a sector boundary and thereby is a multiple of a > sector size in compressed size (ignore trailing garbage). > > That way, you can get by with for example 1 byte of extra > metadata per cluster. Actually the whole "special tables for > compressed filesystem" stuff could be stored at the space > taken by the second FAT :-). Then a compressed filesystem > can not take more space than the uncompressed version even > in the worst case of "all filled with uncompressible files". > > Just some inspiration for the compressed FAT FS dreamers :-D. > > Eric > > > > - > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ > ___ > Freedos-devel mailing list > Freedos-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/freedos-devel > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
[Freedos-devel] compressed FAT filesystems (was: defrag methods)
Hi! > Maybe with adding compression it could make a replacement for > doublespace? (running it through the network redirector obviously) Nope, doublespace compresses the actual data on your disk, not the space which is free anyway ;-). I think it would be nice to have a driver for one of those free / open compressed embedded filesystems for DOS. Some ad-hoc suggestions for a new filesystem: Keep a normal FAT and boot sector and stuff, but wrap all the access to the data clusters. For each data cluster, keep some pointer to where it ACTUALLY starts (granularity: cluster, sector or byte) and allow the clusters to be compressed. You would need some offline "make compressed version of this filesystem" tool and you would not be able to write to the filesystem later. To get a relatively simple implementation, you can assume that each cluster is separately compressed, for example with LZSS. Then you only look at one cluster at a time for uncompressing. Max amount of extra metadata: 32 bits per cluster. Note that you do not store the size of the compressed cluster nor any "this cluster is not compressed flag" - you can easily derive those from the difference between this and next cluster start. To reduce the space taken by the extra metadata, you can use a more "granular" offset for the actual data location, and you can compress the offset array by only storing the size of each compressed cluster instead of its offset. To avoid too much speed loss, you could still keep the offset of every Nth cluster in some array in RAM to reduce the needed "counting". You can even store the size of each compressed cluster in a "granular" way, eg by saying that each compressed cluster has to start at a sector boundary and thereby is a multiple of a sector size in compressed size (ignore trailing garbage). That way, you can get by with for example 1 byte of extra metadata per cluster. Actually the whole "special tables for compressed filesystem" stuff could be stored at the space taken by the second FAT :-). Then a compressed filesystem can not take more space than the uncompressed version even in the worst case of "all filled with uncompressible files". Just some inspiration for the compressed FAT FS dreamers :-D. Eric - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods (OT)
Hi! > For this it is very sad that FreeDOS does not use the LEAN file system > (http://freedos-32.sourceforge.net/doc/lean.html) instead. That has back > pointers for pretty much everything and would as a consequence be much > faster. A lean ad ;-). But you can just as well build data structures with the back pointers in RAM while defrag and similar tools are running. Takes roughly the same amount of RAM as the metadata (FAT, dir) takes on disk. As you often copy those to RAM for work like defragging to get fast access, you basically get some overhead for making the pointers plus you double the amount of RAM that defrag consumes ;-). Note that for example DOSFSCK takes several times the size of your FAT plus several times the size of all directory entries of RAM, too. Actually this would be interesting to know more about: Could some of you run DOSFSCK 2.11c with the -v option on large filesystems, with many files and/or many clusters, and report how much heap it tells you that it has used? It is displayed when DOSFSCK is done. Seems to be 5..10 times the FAT size for systems with few files here... Some "bad case of FAT16" would be interesting (might use 1-5 MB maybe?) as well as, of course, typical FAT32 filesystems. You can find DOSFSCK version 2.11c on my page, as usual: www.coli.uni-saarland.de/~eric/stuff/soft/by-others/dosdosfsck-2.11c.zip (you should already have cwsdpmi anyway, otherwise see e.g. my page) Eric - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Bulgarian keyboard driver for Blocek
On 3/18/08, Ladislav Lacina <[EMAIL PROTECTED]> wrote: > Dimitar Mitov kindly provided the bulgarian keyboard driver for text editor > Blocek. The archives and webpage are updated. > http://www.laaca-mirror.ic.cz > I mirrored this to the ibiblio archive. However, I noticed that the binary rar file hasn't changed from the previous 1.33b. Only the source rar file is different in this new "1.33b". Is that right? I think you forgot to update the binary rar archive. $ ls -l blocek133b* source/blocek133b* -rw-r--r-- 1 freedos users 298505 Mar 18 16:38 blocek133b2.rar -rw-r--r-- 1 freedos users 298505 Nov 3 15:19 blocek133b.rar -rw-r--r-- 1 freedos users 1545674 Mar 18 16:39 source/blocek133b2_s.rar -rw-r--r-- 1 freedos users 190797 Nov 8 19:22 source/blocek133b_s.rar $ md5sum blocek133b* source/blocek133b* d5f72fce7dc546453c37a87b86337728 blocek133b2.rar d5f72fce7dc546453c37a87b86337728 blocek133b.rar ad0b612d35343cdd900bd214d28ef93d source/blocek133b2_s.rar f8e1df6ffcdd6a398a5534ee953458e7 source/blocek133b_s.rar Since you've basically re-released 1.33b, when I mirrored this to ibiblio I used the version code "133b2" (compared to "133b") to indicate that this was a slight update. -jh - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
[Freedos-devel] Bulgarian keyboard driver for Blocek
Dimitar Mitov kindly provided the bulgarian keyboard driver for text editor Blocek. The archives and webpage are updated. http://www.laaca-mirror.ic.cz - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods
Maybe with adding compression it could make a replacement for doublespace? (running it through the network redirector obviously) Imre >- Oorspronkelijk bericht - >Van: Imre Leber [mailto:[EMAIL PROTECTED] >Verzonden: dinsdag, maart 18, 2008 07:10 PM >Aan: freedos-devel@lists.sourceforge.net >Onderwerp: Re: [Freedos-devel] Fwd: freedos defrag methods > > >>- Oorspronkelijk bericht - >>Van: Eric Auer [mailto:[EMAIL PROTECTED] >>Verzonden: dinsdag, maart 18, 2008 01:54 PM >>Aan: freedos-devel@lists.sourceforge.net >>Onderwerp: Re: [Freedos-devel] Fwd: freedos defrag methods >> >> >>Hi! >> >>> I tried to defragment my 10Gb fat32 hdd. I choose quick try. >>> terminates: the working the disk map appears not fragmented. >>> I quit and exit, then restarting defrag: the map is the same >>> as before quick try. >> >>Because quick try only takes care that each file consists of >>only ONE area on the disk. It does not take care that there >>are no unused areas BETWEEN the files... I guess you could do >>a second run after the "quick try" in some mode like "move >>files to the beginning of the disk". Worst case would be that >>this mode has to copy every used data cluster once. This is >>slow but not as slow as having to "move files out of the way". >> >>So combining "defragment files, quick try style" with a 2nd >>step "move files and dirs to the beginning of the disk" after >>that should give you a completely defragmented and compact >>disk (nice for filesystem resize and nice for speed) in most >>cases. In SOME cases, when there is not enough consecutive >>free space for the quick try file defrag, only the very SLOW >>(and more sensitive) full defragmentation in old style could >>help you. >> >>I hope I explained that correctly :-) >> > >Yes, except that move the files to the front of the disk is still way too >slow. Sorry but that is what is has to be. Moving any amount of data on FAT32 >will never work, unless you have a week or so to let the program run. That's >why windows defrag doesn't do this, and that is why we don't do it. > >Just to give an indication of how infinitely slow it is try crunching a FAT16 >volume of some 2GB and then multiply that by a factor of 5 or so. This goes up >very fast if the volume is larger then 2GB. > >Imre > >>Eric >> >> >> >>- >>This SF.net email is sponsored by: Microsoft >>Defy all challenges. Microsoft(R) Visual Studio 2008. >>http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ >>___ >>Freedos-devel mailing list >>Freedos-devel@lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/freedos-devel >> >> > > > >- >This SF.net email is sponsored by: Microsoft >Defy all challenges. Microsoft(R) Visual Studio 2008. >http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ >___ >Freedos-devel mailing list >Freedos-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/freedos-devel > > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods (OT)
For this it is very sad that FreeDOS does not use the LEAN file system (http://freedos-32.sourceforge.net/doc/lean.html) instead. That has back pointers for pretty much everything and would as a consequence be much faster. Voting for a file system change ;-D Imre >- Oorspronkelijk bericht - >Van: Imre Leber [mailto:[EMAIL PROTECTED] >Verzonden: dinsdag, maart 18, 2008 07:10 PM >Aan: freedos-devel@lists.sourceforge.net >Onderwerp: Re: [Freedos-devel] Fwd: freedos defrag methods > > >>- Oorspronkelijk bericht - >>Van: Eric Auer [mailto:[EMAIL PROTECTED] >>Verzonden: dinsdag, maart 18, 2008 01:54 PM >>Aan: freedos-devel@lists.sourceforge.net >>Onderwerp: Re: [Freedos-devel] Fwd: freedos defrag methods >> >> >>Hi! >> >>> I tried to defragment my 10Gb fat32 hdd. I choose quick try. >>> terminates: the working the disk map appears not fragmented. >>> I quit and exit, then restarting defrag: the map is the same >>> as before quick try. >> >>Because quick try only takes care that each file consists of >>only ONE area on the disk. It does not take care that there >>are no unused areas BETWEEN the files... I guess you could do >>a second run after the "quick try" in some mode like "move >>files to the beginning of the disk". Worst case would be that >>this mode has to copy every used data cluster once. This is >>slow but not as slow as having to "move files out of the way". >> >>So combining "defragment files, quick try style" with a 2nd >>step "move files and dirs to the beginning of the disk" after >>that should give you a completely defragmented and compact >>disk (nice for filesystem resize and nice for speed) in most >>cases. In SOME cases, when there is not enough consecutive >>free space for the quick try file defrag, only the very SLOW >>(and more sensitive) full defragmentation in old style could >>help you. >> >>I hope I explained that correctly :-) >> > >Yes, except that move the files to the front of the disk is still way too >slow. Sorry but that is what is has to be. Moving any amount of data on FAT32 >will never work, unless you have a week or so to let the program run. That's >why windows defrag doesn't do this, and that is why we don't do it. > >Just to give an indication of how infinitely slow it is try crunching a FAT16 >volume of some 2GB and then multiply that by a factor of 5 or so. This goes up >very fast if the volume is larger then 2GB. > >Imre > >>Eric >> >> >> >>- >>This SF.net email is sponsored by: Microsoft >>Defy all challenges. Microsoft(R) Visual Studio 2008. >>http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ >>___ >>Freedos-devel mailing list >>Freedos-devel@lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/freedos-devel >> >> > > > >- >This SF.net email is sponsored by: Microsoft >Defy all challenges. Microsoft(R) Visual Studio 2008. >http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ >___ >Freedos-devel mailing list >Freedos-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/freedos-devel > > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods
>- Oorspronkelijk bericht - >Van: Eric Auer [mailto:[EMAIL PROTECTED] >Verzonden: dinsdag, maart 18, 2008 01:54 PM >Aan: freedos-devel@lists.sourceforge.net >Onderwerp: Re: [Freedos-devel] Fwd: freedos defrag methods > > >Hi! > >> I tried to defragment my 10Gb fat32 hdd. I choose quick try. >> terminates: the working the disk map appears not fragmented. >> I quit and exit, then restarting defrag: the map is the same >> as before quick try. > >Because quick try only takes care that each file consists of >only ONE area on the disk. It does not take care that there >are no unused areas BETWEEN the files... I guess you could do >a second run after the "quick try" in some mode like "move >files to the beginning of the disk". Worst case would be that >this mode has to copy every used data cluster once. This is >slow but not as slow as having to "move files out of the way". > >So combining "defragment files, quick try style" with a 2nd >step "move files and dirs to the beginning of the disk" after >that should give you a completely defragmented and compact >disk (nice for filesystem resize and nice for speed) in most >cases. In SOME cases, when there is not enough consecutive >free space for the quick try file defrag, only the very SLOW >(and more sensitive) full defragmentation in old style could >help you. > >I hope I explained that correctly :-) > Yes, except that move the files to the front of the disk is still way too slow. Sorry but that is what is has to be. Moving any amount of data on FAT32 will never work, unless you have a week or so to let the program run. That's why windows defrag doesn't do this, and that is why we don't do it. Just to give an indication of how infinitely slow it is try crunching a FAT16 volume of some 2GB and then multiply that by a factor of 5 or so. This goes up very fast if the volume is larger then 2GB. Imre >Eric > > > >- >This SF.net email is sponsored by: Microsoft >Defy all challenges. Microsoft(R) Visual Studio 2008. >http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ >___ >Freedos-devel mailing list >Freedos-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/freedos-devel > > - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods
Hi! > I tried to defragment my 10Gb fat32 hdd. I choose quick try. > terminates: the working the disk map appears not fragmented. > I quit and exit, then restarting defrag: the map is the same > as before quick try. Because quick try only takes care that each file consists of only ONE area on the disk. It does not take care that there are no unused areas BETWEEN the files... I guess you could do a second run after the "quick try" in some mode like "move files to the beginning of the disk". Worst case would be that this mode has to copy every used data cluster once. This is slow but not as slow as having to "move files out of the way". So combining "defragment files, quick try style" with a 2nd step "move files and dirs to the beginning of the disk" after that should give you a completely defragmented and compact disk (nice for filesystem resize and nice for speed) in most cases. In SOME cases, when there is not enough consecutive free space for the quick try file defrag, only the very SLOW (and more sensitive) full defragmentation in old style could help you. I hope I explained that correctly :-) Eric - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Fwd: freedos defrag methods
I'm tried to defragment my 10Gb hdd fat32 .I choose quick try. terminate the working the disk map appears not fragmented. I quit and exit, then restarting defrag: the map it's the same as before quick try. Why? Imre Leber wrote: > > > Here is the explanation: > > The old methods start by going through the disk from start to finish. For > every cluster it determines which file it has to put there. The file that > is in the way is moved to a different location and the file that should be > there is put in that location. This is very slow. Moreover since much more > data is copied, it tends to be sensitive to disk corruption (resetting the > computer is a very bad idea). > > The new methods, go through the directory tree once, determining which > files are fragmented. For every fragmented file it looks for a location > where it can put it continously. Then it puts the file there. Result: very > fast and far less copying. Moreover data that does not need to be copied > stays where it is. Drawback: on some systems where the disk may be too > full, it might not be able to defragment all files (hence: quick try). > > Imre > > > > - Oorspronkelijk bericht - > Van: Eric Auer [mailto:[EMAIL PROTECTED] > Verzonden: maandag, maart 17, 2008 01:56 PM > Aan: 'Imre Leber' > Onderwerp: Re: freedos defrag methods > > > Hi! > >> Well the old methods from MSDOS are still >> supported for historical reasons. > > I think for smaller partitions (FAT16), they are still useful... > >> But anybody serious about defragmentation should use the >> new "quick try" methods. > > Can you give some more explanation about the pros and cons > of the "quick try" methods? > >> That is the only sane and safe way to do it. > > No method should be unsafe - which have unsafe implementations? > > Eric > > > > > > > > - > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ > ___ > Freedos-devel mailing list > Freedos-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/freedos-devel > > -- View this message in context: http://www.nabble.com/Fwd%3A--freedos-defrag-methods-tp16119594p16120340.html Sent from the FreeDOS - Dev mailing list archive at Nabble.com. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
[Freedos-devel] Fwd: freedos defrag methods
Here is the explanation: The old methods start by going through the disk from start to finish. For every cluster it determines which file it has to put there. The file that is in the way is moved to a different location and the file that should be there is put in that location. This is very slow. Moreover since much more data is copied, it tends to be sensitive to disk corruption (resetting the computer is a very bad idea). The new methods, go through the directory tree once, determining which files are fragmented. For every fragmented file it looks for a location where it can put it continously. Then it puts the file there. Result: very fast and far less copying. Moreover data that does not need to be copied stays where it is. Drawback: on some systems where the disk may be too full, it might not be able to defragment all files (hence: quick try). Imre - Oorspronkelijk bericht - Van: Eric Auer [mailto:[EMAIL PROTECTED] Verzonden: maandag, maart 17, 2008 01:56 PM Aan: 'Imre Leber' Onderwerp: Re: freedos defrag methods Hi! > Well the old methods from MSDOS are still > supported for historical reasons. I think for smaller partitions (FAT16), they are still useful... > But anybody serious about defragmentation should use the > new "quick try" methods. Can you give some more explanation about the pros and cons of the "quick try" methods? > That is the only sane and safe way to do it. No method should be unsafe - which have unsafe implementations? Eric - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
