Re: [Freedos-devel] Interim Build Delayed
Trixter has an 8088 version of LZ4 which is typically faster PKZIP, zlib/gzip, and bzip2. http://www.oldskool.org/pc/lz4_8088 On Fri, Sep 1, 2023 at 11:58 AM Ralf Quint via Freedos-devel < freedos-devel@lists.sourceforge.net> wrote: > On 9/1/2023 7:49 AM, Jerome Shidel via Freedos-devel wrote: > > Hi All, > > > > The FreeDOS Monthly Interim Build for September will be delayed. > > > > The latest version of FDISK includes a large language translation file. > That file is preventing the 720k Floppy version from building. > > As I mentioned in the online meeting, just don't put any language files > on the floppy version in the first place > > > Ralf > > > > > ___ > Freedos-devel mailing list > Freedos-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/freedos-devel > ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
On 9/1/2023 7:49 AM, Jerome Shidel via Freedos-devel wrote: Hi All, The FreeDOS Monthly Interim Build for September will be delayed. The latest version of FDISK includes a large language translation file. That file is preventing the 720k Floppy version from building. As I mentioned in the online meeting, just don't put any language files on the floppy version in the first place Ralf ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
Thanks, I am currently researching which compression algorithm to use, under following conditions: a) running (reasonably fast) on a 8086 with little RAM b) small code size, easy to implement / understand c) decent compression ratio for text I plan to compress the individual help files inside the AMB container [1]. Compressing the individual files and not the whole container guarantees fast seeking. First, deflate [2] came into my mind, which is basically LZSS + Huffman. But, for sake of simplicity, I wanted to test if one of the two steps could be left out without massively impacting the compression ratio. The algorithm you mention seems to implement exactly this, some LSZZ derivative while skipping the Huffman coding. I will have a detailed look at it, but tend to make / re-use a C implementation, so it can be included in the official AMB, if Matheusz is interested. Thanks for the hint! Bernd [1] http://amb.osdn.io/phpamb.php?fname=archiwum/format-20201216.amb [2] https://datatracker.ietf.org/doc/html/rfc1951 On 01.09.2023 20:00, C. Masloch via Freedos-devel wrote: On at 2023-09-01 19:03 +0200, Bernd Böckmann via Freedos-devel wrote: Well, I think it's me to blame ;-) I will try to build some compression mechanism into AMB, so that the help files get smaller. The main FreeDOS help file would also benefit from that, I think. Bernd I recently added one of my executable compression formats to my debugger's online help, allowing it to "assemble" (build) the help pages at build time and then compress them before inclusion (by NASM incbin directive) into the main assembly pass. Perhaps you want to copy parts of this. I used the heatshrink library, "A data compression/decompression library for embedded/real-time systems." [1]. It is available under the ISC License [2]. I created two different depackers, both under the Fair License (but feel free to use and copy these under any usage conditions). One is the executable depacker, which can deal with data (both compressed and decompressed) exceeding 64 KiB, using 8086 segmented addressing [3]. The other is a bit smaller and assumes that all data fits in a single segment each, though the source and destination can be in different segments [4]. For the recent lDebug release 6 I did not yet enable these build options, although everything is implemented for the in-memory online help to be largely compressed. I'm compressing each help page as its own data block, unrelated to one another. Despite this, almost all of the data does compress down by 20% to 60%. (In heatshrink use, the percentage indicates how much of the uncompressed size is dropped to reach the compressed size.) The depacker needs about 300 bytes in my application (for only the memory-to-memory depacker). The compression is done in a loop at build time [5]. I try every -w parameter from 10 to 14 and every -l parameter from 4 to 14 (but -l must be below -w), and use the best result. (Usually -w 10 -l 4.) (I just found that the valid ranges for the parameters are actually larger than this, still testing how well those work.) I use a slightly modified compressed file format, in which the first byte is the -w parameter and the second byte is the -l parameter, and then the original compressed data file is appended. The depacker needs to know the packed size to know when to end depacking. In my debugger's depacker, I append a NUL byte at the end for the user to know how long the unpacked data is, but this could be modified easily to return the size. Regards, ecm [1]: https://github.com/atomicobject/heatshrink [2]: https://github.com/atomicobject/heatshrink/blob/7d419e1fa4830d0b919b9b6a91fe2fb786cf3280/LICENSE [3]: https://hg.pushbx.org/ecm/inicomp/file/41860de1db0e/heatshr.asm [4]: https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/hshrink.asm#l208 [5]: https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/mak.sh#l309 ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
Hi Jerome, hi Bernd, I just checked too, as anyhow it is also my baby (I tested a lot). Fdisk.lng is the replacement for fdisk.en, fdisk.de + other languages in one file. Reason: Space reduction in comparison to old versions, but a space waster on a diskette. It is not needed on a diskette as fdisk falls back to english automatically if the lng. file does not exist. Same like with new htmlhelp. Tested several times. So you can remove the .lng file from diskette. Willi Sent: Friday, September 01, 2023 at 7:25 PM From: "Jerome Shidel via Freedos-devel" To: "Technical discussion and questions for FreeDOS developers." Cc: "Jerome Shidel" Subject: Re: [Freedos-devel] Interim Build Delayed Hi Bernd, > On Sep 1, 2023, at 1:04 PM, Bernd Böckmann via Freedos-devel wrote: > > Well, I think it's me to blame ;-) > > I will try to build some compression mechanism into AMB, so that the help files get smaller. The main FreeDOS help file would also benefit from that, I think. > > Bernd If you look at the 720k boot floppy from T2308, I think there is about 29k free on the diskette. FDISK.EXE went from 69k (I think) to 39k EXE + a 89k FDISK.LNG. Increasing by about 40k. It seems to work fine without the LNG file. So once I can get to figuring out why the RBE is not excluding the LNG file like it is told (probably file name case, dir or something simple), it will fit in the small floppy. But… Compression help files would probably be a good thing as well. Jerome > >> On 01.09.2023 16:49, Jerome Shidel via Freedos-devel wrote: >> Hi All, >> >> The FreeDOS Monthly Interim Build for September will be delayed. >> >> The latest version of FDISK includes a large language translation file. That file is preventing the 720k Floppy version from building. Normally, I would just tell the RBE to exclude that file. There are a number of such exclusions already for files that are not required on the install media. However for some reason, the file is not being excluded when told and the build runs out of space and fails. >> >> Unfortunately, I am extremely busy with other things unrelated to FreeDOS at present. So, I do not know how long it will be until I find time to resolve this issue. It has been quite a while since I’ve done any work on the RBE. So, it may take a little while to figure out why the file is not being removed. >> >> I could just have the RBE forgo building the 720k media. But, I feel it would be better to correct the problem. >> >> I will get to it as soon as I can. But, probably not for several days. >> >> Jerome >> >> ___ >> Freedos-devel mailing list >> Freedos-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/freedos-devel > > > ___ > Freedos-devel mailing list > Freedos-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/freedos-devel ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
On at 2023-09-01 19:03 +0200, Bernd Böckmann via Freedos-devel wrote: Well, I think it's me to blame ;-) I will try to build some compression mechanism into AMB, so that the help files get smaller. The main FreeDOS help file would also benefit from that, I think. Bernd I recently added one of my executable compression formats to my debugger's online help, allowing it to "assemble" (build) the help pages at build time and then compress them before inclusion (by NASM incbin directive) into the main assembly pass. Perhaps you want to copy parts of this. I used the heatshrink library, "A data compression/decompression library for embedded/real-time systems." [1]. It is available under the ISC License [2]. I created two different depackers, both under the Fair License (but feel free to use and copy these under any usage conditions). One is the executable depacker, which can deal with data (both compressed and decompressed) exceeding 64 KiB, using 8086 segmented addressing [3]. The other is a bit smaller and assumes that all data fits in a single segment each, though the source and destination can be in different segments [4]. For the recent lDebug release 6 I did not yet enable these build options, although everything is implemented for the in-memory online help to be largely compressed. I'm compressing each help page as its own data block, unrelated to one another. Despite this, almost all of the data does compress down by 20% to 60%. (In heatshrink use, the percentage indicates how much of the uncompressed size is dropped to reach the compressed size.) The depacker needs about 300 bytes in my application (for only the memory-to-memory depacker). The compression is done in a loop at build time [5]. I try every -w parameter from 10 to 14 and every -l parameter from 4 to 14 (but -l must be below -w), and use the best result. (Usually -w 10 -l 4.) (I just found that the valid ranges for the parameters are actually larger than this, still testing how well those work.) I use a slightly modified compressed file format, in which the first byte is the -w parameter and the second byte is the -l parameter, and then the original compressed data file is appended. The depacker needs to know the packed size to know when to end depacking. In my debugger's depacker, I append a NUL byte at the end for the user to know how long the unpacked data is, but this could be modified easily to return the size. Regards, ecm [1]: https://github.com/atomicobject/heatshrink [2]: https://github.com/atomicobject/heatshrink/blob/7d419e1fa4830d0b919b9b6a91fe2fb786cf3280/LICENSE [3]: https://hg.pushbx.org/ecm/inicomp/file/41860de1db0e/heatshr.asm [4]: https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/hshrink.asm#l208 [5]: https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/mak.sh#l309 ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
Hi Bernd, > On Sep 1, 2023, at 1:04 PM, Bernd Böckmann via Freedos-devel > wrote: > > Well, I think it's me to blame ;-) > > I will try to build some compression mechanism into AMB, so that the help > files get smaller. The main FreeDOS help file would also benefit from that, I > think. > > Bernd If you look at the 720k boot floppy from T2308, I think there is about 29k free on the diskette. FDISK.EXE went from 69k (I think) to 39k EXE + a 89k FDISK.LNG. Increasing by about 40k. It seems to work fine without the LNG file. So once I can get to figuring out why the RBE is not excluding the LNG file like it is told (probably file name case, dir or something simple), it will fit in the small floppy. But… Compression help files would probably be a good thing as well. Jerome > >> On 01.09.2023 16:49, Jerome Shidel via Freedos-devel wrote: >> Hi All, >> >> The FreeDOS Monthly Interim Build for September will be delayed. >> >> The latest version of FDISK includes a large language translation file. That >> file is preventing the 720k Floppy version from building. Normally, I would >> just tell the RBE to exclude that file. There are a number of such >> exclusions already for files that are not required on the install media. >> However for some reason, the file is not being excluded when told and the >> build runs out of space and fails. >> >> Unfortunately, I am extremely busy with other things unrelated to FreeDOS at >> present. So, I do not know how long it will be until I find time to resolve >> this issue. It has been quite a while since I’ve done any work on the RBE. >> So, it may take a little while to figure out why the file is not being >> removed. >> >> I could just have the RBE forgo building the 720k media. But, I feel it >> would be better to correct the problem. >> >> I will get to it as soon as I can. But, probably not for several days. >> >> Jerome >> >> ___ >> Freedos-devel mailing list >> Freedos-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/freedos-devel > > > ___ > Freedos-devel mailing list > Freedos-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/freedos-devel ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
Well, I think it's me to blame ;-) I will try to build some compression mechanism into AMB, so that the help files get smaller. The main FreeDOS help file would also benefit from that, I think. Bernd On 01.09.2023 16:49, Jerome Shidel via Freedos-devel wrote: Hi All, The FreeDOS Monthly Interim Build for September will be delayed. The latest version of FDISK includes a large language translation file. That file is preventing the 720k Floppy version from building. Normally, I would just tell the RBE to exclude that file. There are a number of such exclusions already for files that are not required on the install media. However for some reason, the file is not being excluded when told and the build runs out of space and fails. Unfortunately, I am extremely busy with other things unrelated to FreeDOS at present. So, I do not know how long it will be until I find time to resolve this issue. It has been quite a while since I’ve done any work on the RBE. So, it may take a little while to figure out why the file is not being removed. I could just have the RBE forgo building the 720k media. But, I feel it would be better to correct the problem. I will get to it as soon as I can. But, probably not for several days. Jerome ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
[Freedos-devel] Interim Build Delayed
Hi All, The FreeDOS Monthly Interim Build for September will be delayed. The latest version of FDISK includes a large language translation file. That file is preventing the 720k Floppy version from building. Normally, I would just tell the RBE to exclude that file. There are a number of such exclusions already for files that are not required on the install media. However for some reason, the file is not being excluded when told and the build runs out of space and fails. Unfortunately, I am extremely busy with other things unrelated to FreeDOS at present. So, I do not know how long it will be until I find time to resolve this issue. It has been quite a while since I’ve done any work on the RBE. So, it may take a little while to figure out why the file is not being removed. I could just have the RBE forgo building the 720k media. But, I feel it would be better to correct the problem. I will get to it as soon as I can. But, probably not for several days. Jerome ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel