Thanks, I am currently researching which compression algorithm to use,
under following conditions:
a) running (reasonably fast) on a 8086 with little RAM
b) small code size, easy to implement / understand
c) decent compression ratio for text
I plan to compress the individual help files inside the AMB container
[1]. Compressing the individual files and not the whole container
guarantees fast seeking. First, deflate [2] came into my mind, which is
basically LZSS + Huffman. But, for sake of simplicity, I wanted to test
if one of the two steps could be left out without massively impacting
the compression ratio.
The algorithm you mention seems to implement exactly this, some LSZZ
derivative while skipping the Huffman coding. I will have a detailed
look at it, but tend to make / re-use a C implementation, so it can be
included in the official AMB, if Matheusz is interested.
Thanks for the hint!
Bernd
[1] http://amb.osdn.io/phpamb.php?fname=archiwum/format-20201216.amb
[2] https://datatracker.ietf.org/doc/html/rfc1951
On 01.09.2023 20:00, C. Masloch via Freedos-devel wrote:
On at 2023-09-01 19:03 +0200, Bernd Böckmann via Freedos-devel wrote:
Well, I think it's me to blame ;-)
I will try to build some compression mechanism into AMB, so that the
help files get smaller. The main FreeDOS help file would also benefit
from that, I think.
Bernd
I recently added one of my executable compression formats to my
debugger's online help, allowing it to "assemble" (build) the help
pages at build time and then compress them before inclusion (by NASM
incbin directive) into the main assembly pass. Perhaps you want to
copy parts of this.
I used the heatshrink library, "A data compression/decompression
library for embedded/real-time systems." [1]. It is available under
the ISC License [2]. I created two different depackers, both under the
Fair License (but feel free to use and copy these under any usage
conditions). One is the executable depacker, which can deal with data
(both compressed and decompressed) exceeding 64 KiB, using 8086
segmented addressing [3]. The other is a bit smaller and assumes that
all data fits in a single segment each, though the source and
destination can be in different segments [4].
For the recent lDebug release 6 I did not yet enable these build
options, although everything is implemented for the in-memory online
help to be largely compressed. I'm compressing each help page as its
own data block, unrelated to one another. Despite this, almost all of
the data does compress down by 20% to 60%. (In heatshrink use, the
percentage indicates how much of the uncompressed size is dropped to
reach the compressed size.)
The depacker needs about 300 bytes in my application (for only the
memory-to-memory depacker). The compression is done in a loop at build
time [5]. I try every -w parameter from 10 to 14 and every -l
parameter from 4 to 14 (but -l must be below -w), and use the best
result. (Usually -w 10 -l 4.) (I just found that the valid ranges for
the parameters are actually larger than this, still testing how well
those work.) I use a slightly modified compressed file format, in
which the first byte is the -w parameter and the second byte is the -l
parameter, and then the original compressed data file is appended. The
depacker needs to know the packed size to know when to end depacking.
In my debugger's depacker, I append a NUL byte at the end for the user
to know how long the unpacked data is, but this could be modified
easily to return the size.
Regards,
ecm
[1]: https://github.com/atomicobject/heatshrink
[2]:
https://github.com/atomicobject/heatshrink/blob/7d419e1fa4830d0b919b9b6a91fe2fb786cf3280/LICENSE
[3]: https://hg.pushbx.org/ecm/inicomp/file/41860de1db0e/heatshr.asm
[4]:
https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/hshrink.asm#l208
[5]:
https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/mak.sh#l309
_______________________________________________
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel
_______________________________________________
Freedos-devel mailing list
Freedos-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-devel
