Re: [Freedos-devel] Interim Build Delayed
Hi All, I finally had a little time today to dig into: "Why T2309 build was ignoring the directive to exclude the external FDISK Language file and was running out of space on the 720k boot floppy diskette." After spending an hour or so trying to debug the RBE to locate the problem, the problem finally dawned on me…. The exclusion file list for the Floppy Edition is stored in the FDI-x86 installer project. Along with lists that determine what packages are on the boot media and which packages get installed. I had only added the exclusion directive to just the “master” branch of that project and not the “unstable” branch. So, I just rolled back all the changes I made to the RBE. Then, once I updated the list in the “unstable" project branch for FDI-x86 to exclude the file, the RBE successfully created an Interim Release Build. Since this month is almost over, there is little reason to provide a T2309 build now. So, I’m just going to skip it. We should be good for providing the T2310 Interim Build at the start of October in a few days. Jerome ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
> Thanks, I am currently researching which compression algorithm to use, under > following conditions: > a) running (reasonably fast) on a 8086 with little RAM I know these tiny little things were slow, but I'd still expect 10 KB/sec decompression speed on a 8088; i.e. 'fast enough' (tm). > b) small code size, easy to implement / understand no need to implement/understand (while interesting). there are enough implementations in the wild. upx's LZO would be another candidate. > c) decent compression ratio for text expect compression by 65% or better for text - if the compression unit is large enough. chunks of a few 100 bytes don't compress well. > I plan to compress the individual help files inside the AMB container [1]. > Compressing the individual files and not the whole container guarantees fast > seeking. First, deflate [2] came into my mind, which is basically LZSS + > Huffman. But, for sake of simplicity, I wanted to test if one of the two > steps could be left out without massively impacting the compression ratio. > The algorithm you mention seems to implement exactly this, some LSZZ > derivative while skipping the Huffman coding. I will have a detailed look at > it, but tend to make / re-use a C implementation, so it can be included in > the official AMB, if Matheusz is interested. I'd expect some rather big factor using Huffman because natural language (26 lower case characters, the occasional upper case) have only ~4.5 Bit Entropy for each character. good luck measuring... Tom > Thanks for the hint! > Bernd > [1] http://amb.osdn.io/phpamb.php?fname=archiwum/format-20201216.amb > [2] https://datatracker.ietf.org/doc/html/rfc1951 > On 01.09.2023 20:00, C. Masloch via Freedos-devel wrote: >> On at 2023-09-01 19:03 +0200, Bernd Böckmann via Freedos-devel wrote: >>> Well, I think it's me to blame ;-) >>> >>> I will try to build some compression mechanism into AMB, so that the >> >>> help files get smaller. The main FreeDOS help file would also benefit >> >>> from that, I think. >>> >>> Bernd >> >> I recently added one of my executable compression formats to my > debugger's >> online help, allowing it to "assemble" (build) the help > pages at build >> time and then compress them before inclusion (by NASM > incbin directive) >> into the main assembly pass. Perhaps you want to > copy parts of this. >> >> I used the heatshrink library, "A data compression/decompression > library >> for embedded/real-time systems." [1]. It is available under > the ISC >> License [2]. I created two different depackers, both under the > Fair >> License (but feel free to use and copy these under any usage > conditions). >> One is the executable depacker, which can deal with data > (both compressed >> and decompressed) exceeding 64 KiB, using 8086 > segmented addressing [3]. >> The other is a bit smaller and assumes that > all data fits in a single >> segment each, though the source and > destination can be in different >> segments [4]. >> >> For the recent lDebug release 6 I did not yet enable these build > options, >> although everything is implemented for the in-memory online > help to be >> largely compressed. I'm compressing each help page as its > own data block, >> unrelated to one another. Despite this, almost all of > the data does >> compress down by 20% to 60%. (In heatshrink use, the > percentage indicates >> how much of the uncompressed size is dropped to > reach the compressed size.) >> >> The depacker needs about 300 bytes in my application (for only the > >> memory-to-memory depacker). The compression is done in a loop at build > >> time [5]. I try every -w parameter from 10 to 14 and every -l > parameter >> from 4 to 14 (but -l must be below -w), and use the best > result. (Usually >> -w 10 -l 4.) (I just found that the valid ranges for > the parameters are >> actually larger than this, still testing how well > those work.) I use a >> slightly modified compressed file format, in > which the first byte is the >> -w parameter and the second byte is the -l > parameter, and then the >> original compressed data file is appended. The > depacker needs to know the >> packed size to know when to end depacking. > In my debugger's depacker, I >> append a NUL byte at the end for the user > to know how long the unpacked >> data is, but this could be modified > easily to return the size. >> >> Regards, >> ecm >> >> >> [1]: https://github.com/atomicobject/heatshrink >> [2]: > >> https://github.com/atomicobject/heatshrink/blob/7d419e1fa4830d0b919b9b6a91fe2fb786cf3280/LICENSE >> [3]: https://hg.pushbx.org/ecm/inicomp/file/41860de1db0e/heatshr.asm >> [4]: > >> https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/hshrink.asm#l208 >> [5]: > https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/mak.sh#l309 >> >> >> ___ >> Freedos-devel mailing list >> Freedos-devel@lists.sourceforge.net >>
Re: [Freedos-devel] Interim Build Delayed
Trixter has an 8088 version of LZ4 which is typically faster PKZIP, zlib/gzip, and bzip2. http://www.oldskool.org/pc/lz4_8088 On Fri, Sep 1, 2023 at 11:58 AM Ralf Quint via Freedos-devel < freedos-devel@lists.sourceforge.net> wrote: > On 9/1/2023 7:49 AM, Jerome Shidel via Freedos-devel wrote: > > Hi All, > > > > The FreeDOS Monthly Interim Build for September will be delayed. > > > > The latest version of FDISK includes a large language translation file. > That file is preventing the 720k Floppy version from building. > > As I mentioned in the online meeting, just don't put any language files > on the floppy version in the first place > > > Ralf > > > > > ___ > Freedos-devel mailing list > Freedos-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/freedos-devel > ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
On 9/1/2023 7:49 AM, Jerome Shidel via Freedos-devel wrote: Hi All, The FreeDOS Monthly Interim Build for September will be delayed. The latest version of FDISK includes a large language translation file. That file is preventing the 720k Floppy version from building. As I mentioned in the online meeting, just don't put any language files on the floppy version in the first place Ralf ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
Thanks, I am currently researching which compression algorithm to use, under following conditions: a) running (reasonably fast) on a 8086 with little RAM b) small code size, easy to implement / understand c) decent compression ratio for text I plan to compress the individual help files inside the AMB container [1]. Compressing the individual files and not the whole container guarantees fast seeking. First, deflate [2] came into my mind, which is basically LZSS + Huffman. But, for sake of simplicity, I wanted to test if one of the two steps could be left out without massively impacting the compression ratio. The algorithm you mention seems to implement exactly this, some LSZZ derivative while skipping the Huffman coding. I will have a detailed look at it, but tend to make / re-use a C implementation, so it can be included in the official AMB, if Matheusz is interested. Thanks for the hint! Bernd [1] http://amb.osdn.io/phpamb.php?fname=archiwum/format-20201216.amb [2] https://datatracker.ietf.org/doc/html/rfc1951 On 01.09.2023 20:00, C. Masloch via Freedos-devel wrote: On at 2023-09-01 19:03 +0200, Bernd Böckmann via Freedos-devel wrote: Well, I think it's me to blame ;-) I will try to build some compression mechanism into AMB, so that the help files get smaller. The main FreeDOS help file would also benefit from that, I think. Bernd I recently added one of my executable compression formats to my debugger's online help, allowing it to "assemble" (build) the help pages at build time and then compress them before inclusion (by NASM incbin directive) into the main assembly pass. Perhaps you want to copy parts of this. I used the heatshrink library, "A data compression/decompression library for embedded/real-time systems." [1]. It is available under the ISC License [2]. I created two different depackers, both under the Fair License (but feel free to use and copy these under any usage conditions). One is the executable depacker, which can deal with data (both compressed and decompressed) exceeding 64 KiB, using 8086 segmented addressing [3]. The other is a bit smaller and assumes that all data fits in a single segment each, though the source and destination can be in different segments [4]. For the recent lDebug release 6 I did not yet enable these build options, although everything is implemented for the in-memory online help to be largely compressed. I'm compressing each help page as its own data block, unrelated to one another. Despite this, almost all of the data does compress down by 20% to 60%. (In heatshrink use, the percentage indicates how much of the uncompressed size is dropped to reach the compressed size.) The depacker needs about 300 bytes in my application (for only the memory-to-memory depacker). The compression is done in a loop at build time [5]. I try every -w parameter from 10 to 14 and every -l parameter from 4 to 14 (but -l must be below -w), and use the best result. (Usually -w 10 -l 4.) (I just found that the valid ranges for the parameters are actually larger than this, still testing how well those work.) I use a slightly modified compressed file format, in which the first byte is the -w parameter and the second byte is the -l parameter, and then the original compressed data file is appended. The depacker needs to know the packed size to know when to end depacking. In my debugger's depacker, I append a NUL byte at the end for the user to know how long the unpacked data is, but this could be modified easily to return the size. Regards, ecm [1]: https://github.com/atomicobject/heatshrink [2]: https://github.com/atomicobject/heatshrink/blob/7d419e1fa4830d0b919b9b6a91fe2fb786cf3280/LICENSE [3]: https://hg.pushbx.org/ecm/inicomp/file/41860de1db0e/heatshr.asm [4]: https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/hshrink.asm#l208 [5]: https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/mak.sh#l309 ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
Hi Jerome, hi Bernd, I just checked too, as anyhow it is also my baby (I tested a lot). Fdisk.lng is the replacement for fdisk.en, fdisk.de + other languages in one file. Reason: Space reduction in comparison to old versions, but a space waster on a diskette. It is not needed on a diskette as fdisk falls back to english automatically if the lng. file does not exist. Same like with new htmlhelp. Tested several times. So you can remove the .lng file from diskette. Willi Sent: Friday, September 01, 2023 at 7:25 PM From: "Jerome Shidel via Freedos-devel" To: "Technical discussion and questions for FreeDOS developers." Cc: "Jerome Shidel" Subject: Re: [Freedos-devel] Interim Build Delayed Hi Bernd, > On Sep 1, 2023, at 1:04 PM, Bernd Böckmann via Freedos-devel wrote: > > Well, I think it's me to blame ;-) > > I will try to build some compression mechanism into AMB, so that the help files get smaller. The main FreeDOS help file would also benefit from that, I think. > > Bernd If you look at the 720k boot floppy from T2308, I think there is about 29k free on the diskette. FDISK.EXE went from 69k (I think) to 39k EXE + a 89k FDISK.LNG. Increasing by about 40k. It seems to work fine without the LNG file. So once I can get to figuring out why the RBE is not excluding the LNG file like it is told (probably file name case, dir or something simple), it will fit in the small floppy. But… Compression help files would probably be a good thing as well. Jerome > >> On 01.09.2023 16:49, Jerome Shidel via Freedos-devel wrote: >> Hi All, >> >> The FreeDOS Monthly Interim Build for September will be delayed. >> >> The latest version of FDISK includes a large language translation file. That file is preventing the 720k Floppy version from building. Normally, I would just tell the RBE to exclude that file. There are a number of such exclusions already for files that are not required on the install media. However for some reason, the file is not being excluded when told and the build runs out of space and fails. >> >> Unfortunately, I am extremely busy with other things unrelated to FreeDOS at present. So, I do not know how long it will be until I find time to resolve this issue. It has been quite a while since I’ve done any work on the RBE. So, it may take a little while to figure out why the file is not being removed. >> >> I could just have the RBE forgo building the 720k media. But, I feel it would be better to correct the problem. >> >> I will get to it as soon as I can. But, probably not for several days. >> >> Jerome >> >> ___ >> Freedos-devel mailing list >> Freedos-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/freedos-devel > > > ___ > Freedos-devel mailing list > Freedos-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/freedos-devel ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
On at 2023-09-01 19:03 +0200, Bernd Böckmann via Freedos-devel wrote: Well, I think it's me to blame ;-) I will try to build some compression mechanism into AMB, so that the help files get smaller. The main FreeDOS help file would also benefit from that, I think. Bernd I recently added one of my executable compression formats to my debugger's online help, allowing it to "assemble" (build) the help pages at build time and then compress them before inclusion (by NASM incbin directive) into the main assembly pass. Perhaps you want to copy parts of this. I used the heatshrink library, "A data compression/decompression library for embedded/real-time systems." [1]. It is available under the ISC License [2]. I created two different depackers, both under the Fair License (but feel free to use and copy these under any usage conditions). One is the executable depacker, which can deal with data (both compressed and decompressed) exceeding 64 KiB, using 8086 segmented addressing [3]. The other is a bit smaller and assumes that all data fits in a single segment each, though the source and destination can be in different segments [4]. For the recent lDebug release 6 I did not yet enable these build options, although everything is implemented for the in-memory online help to be largely compressed. I'm compressing each help page as its own data block, unrelated to one another. Despite this, almost all of the data does compress down by 20% to 60%. (In heatshrink use, the percentage indicates how much of the uncompressed size is dropped to reach the compressed size.) The depacker needs about 300 bytes in my application (for only the memory-to-memory depacker). The compression is done in a loop at build time [5]. I try every -w parameter from 10 to 14 and every -l parameter from 4 to 14 (but -l must be below -w), and use the best result. (Usually -w 10 -l 4.) (I just found that the valid ranges for the parameters are actually larger than this, still testing how well those work.) I use a slightly modified compressed file format, in which the first byte is the -w parameter and the second byte is the -l parameter, and then the original compressed data file is appended. The depacker needs to know the packed size to know when to end depacking. In my debugger's depacker, I append a NUL byte at the end for the user to know how long the unpacked data is, but this could be modified easily to return the size. Regards, ecm [1]: https://github.com/atomicobject/heatshrink [2]: https://github.com/atomicobject/heatshrink/blob/7d419e1fa4830d0b919b9b6a91fe2fb786cf3280/LICENSE [3]: https://hg.pushbx.org/ecm/inicomp/file/41860de1db0e/heatshr.asm [4]: https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/hshrink.asm#l208 [5]: https://hg.pushbx.org/ecm/ldebug/file/654bf2cbfffb/source/mak.sh#l309 ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
Hi Bernd, > On Sep 1, 2023, at 1:04 PM, Bernd Böckmann via Freedos-devel > wrote: > > Well, I think it's me to blame ;-) > > I will try to build some compression mechanism into AMB, so that the help > files get smaller. The main FreeDOS help file would also benefit from that, I > think. > > Bernd If you look at the 720k boot floppy from T2308, I think there is about 29k free on the diskette. FDISK.EXE went from 69k (I think) to 39k EXE + a 89k FDISK.LNG. Increasing by about 40k. It seems to work fine without the LNG file. So once I can get to figuring out why the RBE is not excluding the LNG file like it is told (probably file name case, dir or something simple), it will fit in the small floppy. But… Compression help files would probably be a good thing as well. Jerome > >> On 01.09.2023 16:49, Jerome Shidel via Freedos-devel wrote: >> Hi All, >> >> The FreeDOS Monthly Interim Build for September will be delayed. >> >> The latest version of FDISK includes a large language translation file. That >> file is preventing the 720k Floppy version from building. Normally, I would >> just tell the RBE to exclude that file. There are a number of such >> exclusions already for files that are not required on the install media. >> However for some reason, the file is not being excluded when told and the >> build runs out of space and fails. >> >> Unfortunately, I am extremely busy with other things unrelated to FreeDOS at >> present. So, I do not know how long it will be until I find time to resolve >> this issue. It has been quite a while since I’ve done any work on the RBE. >> So, it may take a little while to figure out why the file is not being >> removed. >> >> I could just have the RBE forgo building the 720k media. But, I feel it >> would be better to correct the problem. >> >> I will get to it as soon as I can. But, probably not for several days. >> >> Jerome >> >> ___ >> Freedos-devel mailing list >> Freedos-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/freedos-devel > > > ___ > Freedos-devel mailing list > Freedos-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/freedos-devel ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] Interim Build Delayed
Well, I think it's me to blame ;-) I will try to build some compression mechanism into AMB, so that the help files get smaller. The main FreeDOS help file would also benefit from that, I think. Bernd On 01.09.2023 16:49, Jerome Shidel via Freedos-devel wrote: Hi All, The FreeDOS Monthly Interim Build for September will be delayed. The latest version of FDISK includes a large language translation file. That file is preventing the 720k Floppy version from building. Normally, I would just tell the RBE to exclude that file. There are a number of such exclusions already for files that are not required on the install media. However for some reason, the file is not being excluded when told and the build runs out of space and fails. Unfortunately, I am extremely busy with other things unrelated to FreeDOS at present. So, I do not know how long it will be until I find time to resolve this issue. It has been quite a while since I’ve done any work on the RBE. So, it may take a little while to figure out why the file is not being removed. I could just have the RBE forgo building the 720k media. But, I feel it would be better to correct the problem. I will get to it as soon as I can. But, probably not for several days. Jerome ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
[Freedos-devel] Interim Build Delayed
Hi All, The FreeDOS Monthly Interim Build for September will be delayed. The latest version of FDISK includes a large language translation file. That file is preventing the 720k Floppy version from building. Normally, I would just tell the RBE to exclude that file. There are a number of such exclusions already for files that are not required on the install media. However for some reason, the file is not being excluded when told and the build runs out of space and fails. Unfortunately, I am extremely busy with other things unrelated to FreeDOS at present. So, I do not know how long it will be until I find time to resolve this issue. It has been quite a while since I’ve done any work on the RBE. So, it may take a little while to figure out why the file is not being removed. I could just have the RBE forgo building the 720k media. But, I feel it would be better to correct the problem. I will get to it as soon as I can. But, probably not for several days. Jerome ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
