> Thanks, I am currently researching which compression algorithm to use, under > following conditions:
> a) running (reasonably fast) on a 8086 with little RAM I know these tiny little things were slow, but I'd still expect 10 KB/sec decompression speed on a 8088; i.e. 'fast enough' (tm). > b) small code size, easy to implement / understand no need to implement/understand (while interesting). there are enough implementations in the wild. upx's LZO would be another candidate. > c) decent compression ratio for text expect compression by 65% or better for text - if the compression unit is large enough. chunks of a few 100 bytes don't compress well. > I plan to compress the individual help files inside the AMB container [1]. > Compressing the individual files and not the whole container guarantees fast > seeking. First, deflate [2] came into my mind, which is basically LZSS + > Huffman. But, for sake of simplicity, I wanted to test if one of the two > steps could be left out without massively impacting the compression ratio. > The algorithm you mention seems to implement exactly this, some LSZZ > derivative while skipping the Huffman coding. I will have a detailed look at > it, but tend to make / re-use a C implementation, so it can be included in > the official AMB, if Matheusz is interested. I'd expect some rather big factor using Huffman because natural language (26 lower case characters, the occasional upper case) have only ~4.5 Bit Entropy for each character. good luck measuring... Tom > Thanks for the hint! > Bernd > [1] http://amb.osdn.io/phpamb.php?fname=archiwum/format-20201216.amb > [2] https://datatracker.ietf.org/doc/html/rfc1951 > On 01.09.2023 20:00, C. Masloch via Freedos-devel wrote: >> On at 2023-09-01 19:03 +0200, Bernd Böckmann via Freedos-devel wrote: >>> Well, I think it's me to blame ;-) >>> >>> I will try to build some compression mechanism into AMB, so that the >> >>> help files get smaller. The main FreeDOS help file would also benefit >> >>> from that, I think. >>> >>> Bernd >> >> I recently added one of my executable compression formats to my > debugger's >> online help, allowing it to "assemble" (build) the help > pages at build >> time and then compress them before inclusion (by NASM > incbin directive) >> into the main assembly pass. Perhaps you want to > copy parts of this. >> >> I used the heatshrink library, "A data compression/decompression > library >> for embedded/real-time systems." [1]. It is available under > the ISC >> License [2]. I created two different depackers, both under the > Fair >> License (but feel free to use and copy these under any usage > conditions). >> One is the executable depacker, which can deal with data > (both compressed >> and decompressed) exceeding 64 KiB, using 8086 > segmented addressing [3]. >> The other is a bit smaller and assumes that > all data fits in a single >> segment each, though the source and > destination can be in different >> segments [4]. >> >> For the recent lDebug release 6 I did not yet enable these build > options, >> although everything is implemented for the in-memory online > help to be >> largely compressed. I'm compressing each help page as its > own data block, >> unrelated to one another. Despite this, almost all of > the data does >> compress down by 20% to 60%. (In heatshrink use, the > percentage indicates >> how much of the uncompressed size is dropped to > reach the compressed size.) >> >> The depacker needs about 300 bytes in my application (for only the > >> memory-to-memory depacker). The compression is done in a loop at build > >> time [5]. I try every -w parameter from 10 to 14 and every -l > parameter >> from 4 to 14 (but -l must be below -w), and use the best > result. (Usually >> -w 10 -l 4.) (I just found that the valid ranges for > the parameters are >> actually larger than this, still testing how well > those work.) I use a >> slightly modified compressed file format, in > which the first byte is the >> -w parameter and the second byte is the -l > parameter, and then the >> original compressed data file is appended. The > depacker needs to know the >> packed size to know when to end depacking. > In my debugger's depacker, I >> append a NUL byte at the end for the user > to know how long the unpacked >> data is, but this could be modified > easily to return the size. >> >> Regards, >> ecm >> >> >> [1]: https://github.com/atomicobject/heatshrink >> [2]: > >> https://github.com/atomicobject/heatshrink/blob/7d419e1fa4830d0b919b9b6a91fe2fb786cf3280/LICENSE >> [3]: https://hg.pushbx.org/ecm/inicomp/file/41860de1db0e/heatshr.asm >> [4]: > >> https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/hshrink.asm#l208 >> [5]: > https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/mak.sh#l309 >> >> >> _______________________________________________ >> Freedos-devel mailing list >> Freedos-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/freedos-devel > _______________________________________________ > Freedos-devel mailing list > Freedos-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/freedos-devel Mit freundlichen Grüßen / with kind regards Tom Ehlert +49-15151898538 _______________________________________________ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
