Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik, 

One more thing…

The “workflow” for packages…

I download, restructure, compile, etc. the software.

It then gets staged in completely uncompressed package format as a project at 
https://gitlab.com/FDOS 
(The  sub-groups & projects there will probable be moving to 
https://gitlab.com/FreeDOS/  in the near future)

Once that is done, I zip it up using a older script I wrote a while back. 
(eventually, I’ll add support to do it to the fdvcs.sh script)

Then upload that to my personal repository.

The repo management utilities run on it hourly and will tweak a couple meta 
data fields and place in it’s repo at
https://fd.lod.bz/repos/current/pkg-html/ 


Once it appears in that repo. I download the package from my server and upload 
it to ibiblio.

The repo management utilities run on daily. It will see the metadata fields in 
the package are already modified and 
just put it in the official repository at 
https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/repositories/latest/pkg-html/
 


(This ensures that for the packages in both locations (more on my personal 
server) are identical and hash the same)

Then we come to making a release.

The RBE downloads the latest CDROM.ISO from ibiblio (created by the repo 
management utility) and does a bunch of 
stuff to the packages. Including integrating some NLS stuff from the NLS 
project at https://github.com/shidel/fd-nls 

Finally, the RBE zips those up and produces the release media.

Although with the recent addition of package staging at GitLab and several 
other things, I think a lot of the processing
done on packages in the RBE is going to go away and move further up the chain 
to the package prep stage.

:-)

Jerome





___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik,

Also...

> On Nov 16, 2021, at 1:00 PM, Darik Horn  wrote:
> [..]
> This bash script does the same thing.  The file is also in the shared
> folder in case the listserv mangles it here.  I will submit PRs for
> additional work now that I know about the Gitlab account.

FYI, I still need to make some updates to the RBE to account for some changes 
coming in RC5. 

And it could use improvement in several areas and tons of optimization. But, it 
works and I’ve only got
so much time to spend on it, the installers, required tools, etc.

> #!/bin/bash
> # fdrepack.sh: FreeDOS repository repacking script.
> 
> fdrepack ()
> {
>  TMPDIR=$(mktemp -d)
>  pushd "$TMPDIR" >/dev/null
>  unzip -q "$1"
> 
>  if [[ -d 'SOURCE' ]]
>  then
>pushd 'SOURCE' >/dev/null
>for ii in *.7[Zz]
>do
>  if [[ -r "$ii" ]]
>  then
># Force the source package name to uppercase.
>mkdir $(basename "${ii@U}" '.7Z')
>pushd $(basename "${ii@U}" '.7Z') >/dev/null
>7z x "../$ii"
>popd >/dev/null
>rm "$ii"
>  fi
>done
>for ii in *.[Zz][Ii][Pp]
>do
>  if [[ -r "$ii" ]]
>  then
># Force the source package name to uppercase.
>mkdir $(basename "${ii@U}" '.ZIP')
>pushd $(basename "${ii@U}" '.ZIP') >/dev/null
>unzip -q "../$ii"
>popd >/dev/null
>rm "$ii"
>  fi
>done
> 
>for ii in *
>do
>  if pushd "$ii" >/dev/null
>  then
># Unpack and delete old LFN source archives.
>find -maxdepth 1 -type f -iname sources.7z -exec 7z x {} \;
> -exec rm {} \;
>find -maxdepth 1 -type f -iname sources.zip -exec unzip -q {}
> \; -exec rm {} \;
># Using the store method here makes upstream sources solid in
> the package zip.
>zip -0Xoqr "../${ii@U}.ZIP" .
>popd >/dev/null
>rm -rf "$ii"
>  fi
>done
>popd >/dev/null
>  fi
> 
>  # Use InfoZIP here for the -k and -o switches.
>  zip -0Xkoqr "${1}.repack"
>  advzip -k -p -z -3 -i 15 "${1}.repack"
>  mv "${1}.repack" "${1}"
>  popd >/dev/null
>  rm -rf "$TMPDIR"
> }
> 
> export -f fdrepack
> find ~+ -type f -iname \*.zip -exec bash -c 'fdrepack "{}"' \;
> 

Just some notes on the script. Overall, it looks fine. I did not go through 
each switch. But, I assume they are correct.

ibiblio is the official repo, OS download site and software mirror server. It 
is graciously hosted by the University of
North Carolina. Neither 7zip or advzip are installed on that server. Installing 
linux packages is out. However, we can 
and do run some “mission critical” binaries. So, depending on dependencies we 
could run them. 

${var@U} is not supported by bash on ibiblio, nor the version on my Mac (where 
I run a lot of this stuff for prep, testing, etc).
ibiblio does support ${var^^}, however the Mac doesn’t. So, basically for 
compatibility I end up using some functions that
work on both. (yes, I know they don’t support international characters.)

[[ "$(uname)" == "Darwin" ]] && MACOS=true || unset MACOS
UPPER_CHARS='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
LOWER_CHARS='abcdefghijklmnopqrstuvwxyz'

function lowerCase () {
if [[ ${MACOS} ]] ; then
# Slower method not using ${variable,,}
local i c o
for ((i=0;i<${#1};i++)) ; do
c="${1:${i}:1}"
if [[ "${c//[${UPPER_CHARS}]}" != "${c}" ]] ; then
c="${UPPER_CHARS%%${c}*}"
c="${LOWER_CHARS:${#c}:1}"
fi
o="${o}${c}"
done
echo "${o}"
else
echo "${1,,}"
fi
}

function upperCase () {
if [[ ${MACOS} ]] ; then
# Slower method not using ${variable^^}
local i c o
for ((i=0;i<${#1};i++)) ; do
c="${1:${i}:1}"
if [[ "${c//[${LOWER_CHARS}]}" != "${c}" ]] ; then
c="${LOWER_CHARS%%${c}*}"
c="${UPPER_CHARS:${#c}:1}"
fi
o="${o}${c}"
done
echo "${o}"
else
echo "${1^^}"
fi
}

Obviously, they are no where near as fast as the built-in shell versions. 
But generally, are much faster than using tr “[:lower:]” “[:upper:]” and the 
like.
Technically, since using them through a sub-shell like fname=$(upperCase 
“${oname}”) 
it doesn’t really need to declare the vars as local.

Archive timestamps are lost. While technically a new zip file, it may be more 
useful to apply
the old archives timestamp to the new one. It is not difficult. I do something 
like that in the 
fdvcs.sh script (part of the PkgDevKit, another work in progress). It is just 
something that
we will need to consider.

While it would be good to reduce the sizes of the zip archives, there is 
currently plenty of 
room on the different release media. At present, I have several dozen critical 
things that
need done for RC5. After RC5, I will look into it more.

One area that could use volunteers now is package updates. There are roughly as 
dozen
BASE packages 

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik,

> On Nov 16, 2021, at 1:01 PM, Darik Horn  wrote:
> 
> Jerome,
> 
>> First, the new versions would need to be checked to ensure compatibility. 
>> They would probably be fine. But, testing would still need done. I honestly 
>> just done have the time for that at present.
> 
> This work is, in part, the result of doing coverage testing on the 
> distribution.
> 
> 
>> But, I think it is to late to consider such a large unplanned change this 
>> close to the release 1.3-RC5.
> 
> Note that this fixes the fails-to-install-all-packages-with-source bug
> in the latest release candidate.

No. 

The package for HTMLHELP used a source directory as HELP which conflicted with 
another package. I’ve corrected the package on the repository. So, there is no 
longer a conflict.

>> I appreciate the effort you put into creating a PowerShell script to repack 
>> the repository. However, I do not know if we are going to do that.
> 
> This bash script does the same thing.  The file is also in the shared
> folder in case the listserv mangles it here.  I will submit PRs for
> additional work now that I know about the Gitlab account.
> 
> 
> #!/bin/bash
> # fdrepack.sh: FreeDOS repository repacking script.
> 
> fdrepack ()
> {
>  TMPDIR=$(mktemp -d)
>  pushd "$TMPDIR" >/dev/null
>  unzip -q "$1"
> 
>  if [[ -d 'SOURCE' ]]
>  then
>pushd 'SOURCE' >/dev/null
>for ii in *.7[Zz]
>do
>  if [[ -r "$ii" ]]
>  then
># Force the source package name to uppercase.
>mkdir $(basename "${ii@U}" '.7Z')
>pushd $(basename "${ii@U}" '.7Z') >/dev/null
>7z x "../$ii"
>popd >/dev/null
>rm "$ii"
>  fi
>done
>for ii in *.[Zz][Ii][Pp]
>do
>  if [[ -r "$ii" ]]
>  then
># Force the source package name to uppercase.
>mkdir $(basename "${ii@U}" '.ZIP')
>pushd $(basename "${ii@U}" '.ZIP') >/dev/null
>unzip -q "../$ii"
>popd >/dev/null
>rm "$ii"
>  fi
>done
> 
>for ii in *
>do
>  if pushd "$ii" >/dev/null
>  then
># Unpack and delete old LFN source archives.
>find -maxdepth 1 -type f -iname sources.7z -exec 7z x {} \;
> -exec rm {} \;
>find -maxdepth 1 -type f -iname sources.zip -exec unzip -q {}
> \; -exec rm {} \;
># Using the store method here makes upstream sources solid in
> the package zip.
>zip -0Xoqr "../${ii@U}.ZIP" .
>popd >/dev/null
>rm -rf "$ii"
>  fi
>done
>popd >/dev/null
>  fi
> 
>  # Use InfoZIP here for the -k and -o switches.
>  zip -0Xkoqr "${1}.repack"
>  advzip -k -p -z -3 -i 15 "${1}.repack"
>  mv "${1}.repack" "${1}"
>  popd >/dev/null
>  rm -rf "$TMPDIR"
> }
> 
> export -f fdrepack
> find ~+ -type f -iname \*.zip -exec bash -c 'fdrepack "{}"' \;
> 
> 
> ___
> Freedos-user mailing list
> Freedos-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/freedos-user



___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Darik Horn]

Jerome,

> First, the new versions would need to be checked to ensure compatibility. 
> They would probably be fine. But, testing would still need done. I honestly 
> just done have the time for that at present.

This work is, in part, the result of doing coverage testing on the distribution.


> But, I think it is to late to consider such a large unplanned change this 
> close to the release 1.3-RC5.

Note that this fixes the fails-to-install-all-packages-with-source bug
in the latest release candidate.


> I appreciate the effort you put into creating a PowerShell script to repack 
> the repository. However, I do not know if we are going to do that.

This bash script does the same thing.  The file is also in the shared
folder in case the listserv mangles it here.  I will submit PRs for
additional work now that I know about the Gitlab account.


#!/bin/bash
# fdrepack.sh: FreeDOS repository repacking script.

fdrepack ()
{
  TMPDIR=$(mktemp -d)
  pushd "$TMPDIR" >/dev/null
  unzip -q "$1"

  if [[ -d 'SOURCE' ]]
  then
pushd 'SOURCE' >/dev/null
for ii in *.7[Zz]
do
  if [[ -r "$ii" ]]
  then
# Force the source package name to uppercase.
mkdir $(basename "${ii@U}" '.7Z')
pushd $(basename "${ii@U}" '.7Z') >/dev/null
7z x "../$ii"
popd >/dev/null
rm "$ii"
  fi
done
for ii in *.[Zz][Ii][Pp]
do
  if [[ -r "$ii" ]]
  then
# Force the source package name to uppercase.
mkdir $(basename "${ii@U}" '.ZIP')
pushd $(basename "${ii@U}" '.ZIP') >/dev/null
unzip -q "../$ii"
popd >/dev/null
rm "$ii"
  fi
done

for ii in *
do
  if pushd "$ii" >/dev/null
  then
# Unpack and delete old LFN source archives.
find -maxdepth 1 -type f -iname sources.7z -exec 7z x {} \;
-exec rm {} \;
find -maxdepth 1 -type f -iname sources.zip -exec unzip -q {}
\; -exec rm {} \;
# Using the store method here makes upstream sources solid in
the package zip.
zip -0Xoqr "../${ii@U}.ZIP" .
popd >/dev/null
rm -rf "$ii"
  fi
done
popd >/dev/null
  fi

  # Use InfoZIP here for the -k and -o switches.
  zip -0Xkoqr "${1}.repack"
  advzip -k -p -z -3 -i 15 "${1}.repack"
  mv "${1}.repack" "${1}"
  popd >/dev/null
  rm -rf "$TMPDIR"
}

export -f fdrepack
find ~+ -type f -iname \*.zip -exec bash -c 'fdrepack "{}"' \;


___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Eric Auer]

Hi Darik and Jerome,

> * The 1.44 MB floppy set is reduced from eight to three diskettes.
> * The 1.20 MB floppy set is reduced from ten to four diskettes.
> * The 720 KB floppy set is reduced from fifteen to six diskettes.

That is a huge difference and RAR is not overly exotic. I also
would like to point out that infozip also comes with zipsplit,
so having archives spanning multiple disks is nothing strange
for the well-established packers, including ZIP.

> Creating 360 KB floppies or smaller would require a big extension of
> the installation program.

To be honest, people who have only 360k drives are unlikely to have
any harddisk where you could install to. The one workstation type
PC which had both (floppy limited to 360k and a harddisk) that I
know about was too luxurious to bother to be fully PC compatible,
so it would not run FreeDOS, although it could run some of our apps.

So for THAT tiny disk size of 360k, I would rather focus on having
a good way to work directly from floppy without installing.

Preferably optimized for having very little RAM. Probably something
MS DOS 3 style, with some of the modern features stripped away.

> 1a. Replace FREEDOS.SAF with a FREEDOS.TGZ tarball.
> 1b. Use coreutils to "SPLIT FREEDOS.TGZ" across floppy media.
> 1c. Unpack it with a "COPY /B PART1+PART2" method and some batch logic.

As mentioned, that would require extra temp space for the merged
tarball before you unpack it, as DOS has no in-memory pipelines,
but of course for our DISKETTE distro, the idea is to include
only a limited selection of apps, so the tarball size is limited
as well and could be okay. I myself once made a three floppy proof
of concept distro with all BASE packages pre-installed, which had
one archive file with various less important files on the third.

Regards, Eric



___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik,

> On Nov 15, 2021, at 5:08 AM, Darik Horn  wrote:
> 
> Jerome,
> 
>> If you can figure out how to get it to work, then it will be worth 
>> considering.
> 
> It works.  Updated builds are posted here:
> 
> https://drive.google.com/drive/folders/1b7IE-EsK5R1ROmXTaEvYNi9-kfVvH5wi?usp=sharing

I’ll try to look at it later today.

> 
> * The 1.44 MB floppy set is reduced from eight to three diskettes.
> * The 1.20 MB floppy set is reduced from ten to four diskettes.
> * The 720 KB floppy set is reduced from fifteen to six diskettes.
> 
> Creating 360 KB floppies or smaller would require a big extension of
> the installation program.

Yes. The installer needs updated to support smaller disk sizes. Eventually, I 
will add it. It wont be difficult. But, 360s aren’t common compared to the 
larger capacities. Therefore, it isn’t a priority.

> 
> This update also includes forward error correction for bad media.
> 
> The nearby BOOT.diff file describes what changed to make the demo happen.
> 
> 
>> 1) Wether or not it’s License is using an acceptable open source license.
>> 2) Usage is approved. Not up to me. But if licenses are ok, it should not be 
>> a problem.
> 
> The encoder is non-free and zero-cost, but the demo otherwise
> implements all features.

Not being open source will probably be an issue.

> 
> 
>> 3) Can be told to create spans of a specific size (so can be used on all 
>> floppy variations, one size fits all).
> 
> Yes.
> 
> 
>> 4) Can replace SLICER with little to no overhead added to the installer 
>> and/or RBE (Release Build Environment)
>> ...
>> https://github.com/shidel/FDI-x86/tree/master/SETTINGS
> 
> Thanks for posting this Github link, which I will review later.  I
> looked for a FreeDOS build system earlier but didn't find it.

Your welcome. 

The RBE is at https://gitlab.com/FreeDOS/OS/builder

There are also mirrors for FDI & FDI-x86 there as well as some other stuff. 

If your interested in building a FreeDOS release using the RBE, it’s rather 
easy to setup and do. Just follow the wiki for it on GitLab. 

The RBE itself is rather complicated and is a work-in-progress. But it works. 
:-)

On a side note, here is just something to ponder… A typical end user is 
installing FreeDOS into VirtualBox on Windows… So… I sit here running a Mac. 
Using VirtualBox, to run a Linux VM for the RBE. The RBE downloads and 
processes the raw installers and packages. It then dynamically builds some 
custom boot images and runs QEMU instances with FreeDOS. Those instances run 
batch files generated by the RBE to create the different install media. That 
media then is loaded into a different machine by the end user. They then run 
another virtual machine and run the installer for FreeDOS. The installer then 
installed the OS and creates custom config files for that system.

And that was an over-simplification of a release build to install chain. Trying 
to do it by hand would be crazy making and prone to mistakes. 

> 
> 
>> 5) Can provide the same level of flexibility without the need to manually 
>> modify code when there are package/tag changes.
> 
> In the updated demo, a @LISTFILE substitutes each possible "SLICER /G
> %TPU%" invocation.
> 
> 
>> Can it display general text during the install process?
> 
> Yes, this can be scripted through the IDOS.SFX module.  The given demo
> displays progress per-file, which is the default.
> 
> 
>> Does it support NLS and provide text in the users language for prompts?
> 
> Dunno about NLS, but it can do ANSI.SYS style text in if/then/else blocks.
> 
> 
>> On systems that support change-line, can it auto-resume without being told 
>> to continue?
> 
> Dunno what change-line support is...
> 
> But the extractor won't prompt for a media change if it can already
> see the next volume, so floppy contents can be copied to a ZipDrive or
> HDD and the installer will behave properly.
> 
> 
>> After completion, can it just return an errorlevel to the installer and not 
>> stop with an “I’m done, Press OK” message?
> 
> The available %ERRORLEVEL% returns are sensible and actionable.
> 
> 
>> Those and other settings are available in the projects SETTINGS directory. 
>> Specifically, the X86_ALL.LST defines what gets what slicer tags for 
>> assembling the archive.
> 
> This RAR demo is somewhat an accident and I appreciate how licensing
> matters.  Two alternatives to using RAR or finishing SLICER come to
> mind now that I better understand the solution:
> 
> 1a. Replace FREEDOS.SAF with a FREEDOS.TGZ tarball.
> 1b. Use coreutils to "SPLIT FREEDOS.TGZ" across floppy media.
> 1c. Unpack it with a "COPY /B PART1+PART2" method and some batch logic.
> 
> This is how Linux distributions did installation before CD-R media was
> invented.  Zero new compiled code is required to implement it for
> FreeDOS, and it will cost 10 MB of temporary hard disk space at
> install time.
> 
> 2. Use the GPL licensing option in unrarlib to create a RAR2 encoder.
> Make a "grar" 

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik,

I appreciate the effort you put into creating a PowerShell script to repack the 
repository. However, I do not know if we are going to do that. 

First, the new versions would need to be checked to ensure compatibility. They 
would probably be fine. But, testing would still need done. I honestly just 
done have the time for that at present.

Second, there is no way you could’ve known this. But, none of the machines 
(virtual or real) run Windows. Mostly Linux, some DOS and macOS (unix/BSD). 

There are numerous utilities and scripts we use that operate on all packages in 
the repo and elsewhere. So, whipping up something to run through them and 
recompress, is not too difficult.

But, I think it is to late to consider such a large unplanned change this close 
to the release 1.3-RC5. 

Once RC5 is released, I will look into it some and possibly add some 
recompression into the tool chain. 

:-)

Jerome


___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Darik Horn]

Jerome,

> If you can figure out how to get it to work, then it will be worth 
> considering.

It works.  Updated builds are posted here:

https://drive.google.com/drive/folders/1b7IE-EsK5R1ROmXTaEvYNi9-kfVvH5wi?usp=sharing

* The 1.44 MB floppy set is reduced from eight to three diskettes.
* The 1.20 MB floppy set is reduced from ten to four diskettes.
* The 720 KB floppy set is reduced from fifteen to six diskettes.

Creating 360 KB floppies or smaller would require a big extension of
the installation program.

This update also includes forward error correction for bad media.

The nearby BOOT.diff file describes what changed to make the demo happen.


> 1) Wether or not it’s License is using an acceptable open source license.
> 2) Usage is approved. Not up to me. But if licenses are ok, it should not be 
> a problem.

The encoder is non-free and zero-cost, but the demo otherwise
implements all features.


> 3) Can be told to create spans of a specific size (so can be used on all 
> floppy variations, one size fits all).

Yes.


> 4) Can replace SLICER with little to no overhead added to the installer 
> and/or RBE (Release Build Environment)
> ...
>  https://github.com/shidel/FDI-x86/tree/master/SETTINGS

Thanks for posting this Github link, which I will review later.  I
looked for a FreeDOS build system earlier but didn't find it.


> 5) Can provide the same level of flexibility without the need to manually 
> modify code when there are package/tag changes.

In the updated demo, a @LISTFILE substitutes each possible "SLICER /G
%TPU%" invocation.


> Can it display general text during the install process?

Yes, this can be scripted through the IDOS.SFX module.  The given demo
displays progress per-file, which is the default.


> Does it support NLS and provide text in the users language for prompts?

Dunno about NLS, but it can do ANSI.SYS style text in if/then/else blocks.


> On systems that support change-line, can it auto-resume without being told to 
> continue?

Dunno what change-line support is...

But the extractor won't prompt for a media change if it can already
see the next volume, so floppy contents can be copied to a ZipDrive or
HDD and the installer will behave properly.


> After completion, can it just return an errorlevel to the installer and not 
> stop with an “I’m done, Press OK” message?

The available %ERRORLEVEL% returns are sensible and actionable.


> Those and other settings are available in the projects SETTINGS directory. 
> Specifically, the X86_ALL.LST defines what gets what slicer tags for 
> assembling the archive.

This RAR demo is somewhat an accident and I appreciate how licensing
matters.  Two alternatives to using RAR or finishing SLICER come to
mind now that I better understand the solution:

1a. Replace FREEDOS.SAF with a FREEDOS.TGZ tarball.
1b. Use coreutils to "SPLIT FREEDOS.TGZ" across floppy media.
1c. Unpack it with a "COPY /B PART1+PART2" method and some batch logic.

This is how Linux distributions did installation before CD-R media was
invented.  Zero new compiled code is required to implement it for
FreeDOS, and it will cost 10 MB of temporary hard disk space at
install time.

2. Use the GPL licensing option in unrarlib to create a RAR2 encoder.
Make a "grar" utility like there is "gzip", which would have general
application, especially in the retro computing.


___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Darik Horn]

Jerome,

This PowerShell script generates the optimization that I want as an
end-user on the 'freedos/files/repositories/latest' tree.  Only 7Z.exe
is used because 7-Zip 19.00 and AdvZip 2.1 generate outputs that are
nearly identical for a full repack.

# FDREPACK.PS1: FreeDOS repository repacking script.
$ZIPS = Get-ChildItem -Recurse -File -Filter *.ZIP

ForEach ($ZIP in $ZIPS) {
  $TMPDIR = Join-Path $ENV:TMP -ChildPath $('FDREPACK_' +
$ZIP.BaseName + '_' + $(New-GUID))
  $TMPDIR = New-Item -ItemType Directory -Path $TMPDIR
  Push-Location $TMPDIR
  7Z x $ZIP.FullName
  If (Test-Path -Path 'SOURCE' -PathType Container) {
Push-Location 'SOURCE'
ForEach ($II in @(Get-ChildItem -File -Filter '*.7Z';
Get-ChildItem -File -Filter '*.ZIP')) {
NewItem -ItemType Directory -Path $II.BaseName
Push-Location -Path $II.BaseName
7Z x $II.FullName
Pop-Location
Remove-Item $II.FullName
  }
  ForEach ($JJ in Get-ChildItem -Directory) {
Push-Location $JJ
If (Test-Path -Path 'SOURCES.7Z' -PathType Leaf) {
  7z x 'SOURCES.7Z'
  Remove-Item 'SOURCES.7Z'
}
If (Test-Path -Path 'SOURCES.ZIP' -PathType Leaf) {
  7z x 'SOURCES.ZIP'
  Remove-Item 'SOURCES.ZIP'
}
  ATTRIB -A /S
  # This makes upstream sources solid in the distro package.
  7Z a -tzip -mm=copy -mtc=off -stl $('..\' + $JJ + '.ZIP')
  Pop-Location
  Remove-Item -Recurse $JJ
}
Pop-Location # SOURCE
  }
  ATTRIB -A /S
  # These are the best zip format 2.0 parameters at the pkzip-2.0 level.
  7Z a -tzip -mm=deflate -mfb=258 -mpass=15 -mmt=on -mtc=off -stl
$($ZIP.FullName + '.repack')
  Move-Item -Force $($ZIP.FullName + '.repack') ($ZIP.FullName)
  Pop-Location # $TMPDIR
  Remove-Item -Force -Recurse $TMPDIR
}


___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik,

I did not read every entry in that list of files. But there were a couple 
things I noticed while skimming it…

Those files are in the mirrors section of the ibiblio server. They don’t really 
have anything directly to do with the packages in the software repo area. Nor 
do they have anything directly to do with the packages on the release media.

Mirror and Repo are just my names for the different areas on the server. They 
don’t have any official names. Anyhow, they are not really related to each 
other.

I have nothing at all to do with the files in the mirrored area. However, I 
believe they are exact copies of the original mirrored files. 

On the other hand, I manage the Repo area and packages. Nowadays, it requires 
little manual intervention on my part and is fully automated. However, I must 
still throw together packages to upload to that section. That is a tedious 
process. 

When putting together those updates, I usually pull them directly from there 
website and not the mirrors area on ibiblio. I have noticed that several 
developers use different programs to generate their zip downloads and while not 
actually corrupt. The compatibility with those files varies among decompression 
software. 

After all the preparations have been done and the files are placed in the 
proper directories for a package, I then use a script to recompress them. That 
script checks for LFN files. If it finds any in the Source directory, it 
compress the whole thing to preserve the LFNs. As for other LFNs elsewhere in 
the package, those individual files are placed in an LFNFILES.zip. 

After the process of preserving LFNs is complete, the entire package is then 
compressed in DOS compatibility mode.

Anyhow, I can’t say wether or not any files in the mirror section are corrupt 
or just incompatible with that zip program. 

But, as I mentioned in an earlier post, I do know there is at least one that 
contains a corrupt archive nested several layers deep inside it’s package. That 
would be the package for OpenGEM. There may be others.

:-)

Jerome




___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Darik Horn]

Eric,

> Please describe that "normalizing" process which makes
> the source codes inside the distro packages behave like a
> solid archive.

Look at a FreeDOS package like 'latest/apps/doszip.zip' and notice the
embedded SOURCE/SOURCES.ZIP file.

Every package with long file names in upstream source code is like
this. Get the optimization by repacking all FreeDOS packages like they
use LFN.


> Can you provide a list of those damaged ZIP files to us?

$ find -type f -iname  \*.zip -print0 | xargs -0 -n1 advzip --test --pedantic
Failed read end of central directory on
freedos/files/devel/asm/nasm/0.98.37/nasm-0.98.37-xdoc.zip
Invalid local purpose bit flag 0/2 on
freedos/files/devel/asm/octasm/octasm17.zip
Unsupported compression method on file RFM.EXE on
freedos/files/devel/c/bflat/fbug010.zip
Unsupported compression method on file APPINFO/I16BUDOC.LSM on
freedos/files/devel/c/gcc-ia16/20200321/i16budoc.zip
Unsupported compression method on file APPINFO/I16BUTIL.LSM on
freedos/files/devel/c/gcc-ia16/20200321/i16butil.zip
Unsupported compression method on file APPINFO/I16ELKLC.LSM on
freedos/files/devel/c/gcc-ia16/20200321/i16elklc.zip
Unsupported compression method on file APPINFO/I16GCC.LSM on
freedos/files/devel/c/gcc-ia16/20200321/i16gcc.zip
Unsupported compression method on file APPINFO/I16GCDOC.LSM on
freedos/files/devel/c/gcc-ia16/20200321/i16gcdoc.zip
Unsupported compression method on file APPINFO/I16NEWLI.LSM on
freedos/files/devel/c/gcc-ia16/20200321/i16newli.zip
Unsupported compression method on file APPINFO/I16BUDOC.LSM on
freedos/files/devel/c/gcc-ia16/20201106/i16budoc.zip
Unsupported compression method on file APPINFO/I16BUTIL.LSM on
freedos/files/devel/c/gcc-ia16/20201106/i16butil.zip
Unsupported compression method on file APPINFO/I16GCC.LSM on
freedos/files/devel/c/gcc-ia16/20201106/i16gcc.zip
Unsupported compression method on file APPINFO/I16GCDOC.LSM on
freedos/files/devel/c/gcc-ia16/20201106/i16gcdoc.zip
Unsupported compression method on file APPINFO/I16NEWLI.LSM on
freedos/files/devel/c/gcc-ia16/20201106/i16newli.zip
Unsupported compression method on file APPINFO/I16BUDOC.LSM on
freedos/files/devel/c/gcc-ia16/20210305/i16budoc.zip
Unsupported compression method on file APPINFO/I16BUTIL.LSM on
freedos/files/devel/c/gcc-ia16/20210305/i16butil.zip
Unsupported compression method on file APPINFO/I16GCC.LSM on
freedos/files/devel/c/gcc-ia16/20210305/i16gcc.zip
Unsupported compression method on file APPINFO/I16GCDOC.LSM on
freedos/files/devel/c/gcc-ia16/20210305/i16gcdoc.zip
Unsupported compression method on file APPINFO/I16NEWLI.LSM on
freedos/files/devel/c/gcc-ia16/20210305/i16newli.zip
Unsupported compression method on file APPINFO/I16BUDOC.LSM on
freedos/files/devel/c/gcc-ia16/20210418/i16budoc.zip
Unsupported compression method on file APPINFO/I16BUTIL.LSM on
freedos/files/devel/c/gcc-ia16/20210418/i16butil.zip
Unsupported compression method on file APPINFO/I16GCC.LSM on
freedos/files/devel/c/gcc-ia16/20210418/i16gcc.zip
Unsupported compression method on file APPINFO/I16GCDOC.LSM on
freedos/files/devel/c/gcc-ia16/20210418/i16gcdoc.zip
Unsupported compression method on file APPINFO/I16NEWLI.LSM on
freedos/files/devel/c/gcc-ia16/20210418/i16newli.zip
Unsupported compression method on file APPINFO/I16LBI86.LSM on
freedos/files/devel/c/gcc-ia16/libi86/20201201/i16lbi86.zip
Unsupported compression method on file APPINFO/I16LBI86.LSM on
freedos/files/devel/c/gcc-ia16/libi86/20210227/i16lbi86.zip
Unsupported compression method on file APPINFO/I16LBI86.LSM on
freedos/files/devel/c/gcc-ia16/libi86/20210418/i16lbi86.zip
Unsupported compression method on file APPINFO/I16LBI86.LSM on
freedos/files/devel/c/gcc-ia16/libi86/20210915/i16lbi86.zip
Unsupported compression method on file LIB286/MATH87C.LIB on
freedos/files/devel/c/openwatcom/1.5/clib_a16.zip
Unsupported compression method on file LIB286/DOS/CLIBC.LIB on
freedos/files/devel/c/openwatcom/1.5/clib_d16.zip
Unsupported compression method on file LIB286/OS2/CLIBC.LIB on
freedos/files/devel/c/openwatcom/1.5/clib_o16.zip
Unsupported compression method on file LIB386/OS2/CLBRDLL.LIB on
freedos/files/devel/c/openwatcom/1.5/clib_o32.zip
Unsupported compression method on file SAMPLES/CLIBEXAM/ABORT.C on
freedos/files/devel/c/openwatcom/1.5/clib_samples.zip
Unsupported compression method on file LIB286/WIN/CLIBC.LIB on
freedos/files/devel/c/openwatcom/1.5/clib_w16.zip
Unsupported compression method on file LIB386/NT/CLBRDLL.LIB on
freedos/files/devel/c/openwatcom/1.5/clib_w32.zip
Unsupported compression method on file LIB286/MATH87L.LIB on
freedos/files/devel/c/openwatcom/1.5/cm_clib_a16.zip
Unsupported compression method on file LIB386/MATH387R.LIB on
freedos/files/devel/c/openwatcom/1.5/cm_clib_a32.zip
Unsupported compression method on file LIB286/DOS/BINMODE.OBJ on
freedos/files/devel/c/openwatcom/1.5/cm_clib_d16.zip
Unsupported compression method on file BINW/WSTUB.EXE on
freedos/files/devel/c/openwatcom/1.5/cm_clib_d32.zip
Unsupported compression 

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Eric,

> On Nov 11, 2021, at 4:13 PM, E. Auer  wrote:
> 
> 
> Hi! Please describe that "normalizing" process which makes
> the source codes inside the distro packages behave like a
> solid archive. If you remove them from the ZIP and add a
> TAR of the sources to the ZIP instead, you would get that
> type of result, but it would mean that the install process
> will have to be modified significantly. Also, you would
> need temp storage for the TAR because DOS has fake pipes.
> 
>> 2,586,758,742 bytes, original total ZIP size
>> 2,535,458,056 bytes, after `advzip -k -p -z -3 `
> 
> ...
> 
>> But normalizing the source code that is bundled in distro packages is
>> where the big savings happen because it makes the most compressible part of
>> the archive behave like a solid 7Z file.
> 
> ...
> 
>> The advzip output is indeed compatible with both pkunzip-2.04g and
>> pkunzip-2.50 running on a DOS machine with one megabyte of memory.
> 
>> The way AdvanceCOMP checks and retains metadata is satisfying.  If you run
>> advzip on the entire FreeDOS collection, then you'll notice a non-trivial
>> number of malformed and/or corrupt ZIP files.
> 
> You mean they were already malformed before you recompressed them?
> Can you provide a list of those damaged ZIP files to us?

I think I’ve mentioned that before. Either on the mailing list or during one of 
the online get-togethers.

When the RBE is building metadata about the packages and the release in 
general, it verifies the zip archives. It also, dives through nested zip 
archives. The RBE does throw non-critical error messages/logs about corrupt 
zips. 

There are corrupt zip archives inside some packages. I’m on my phone right now. 
So, I cannot recall which or how many. Next time I do a test build with the RBE 
I’ll let you know.

If I recall correctly, at least one is a couple zips deep in the OpenGEM 
sources.



> 
> Thanks! Regards, Eric
> 
> 
> 
> 
> ___
> Freedos-user mailing list
> Freedos-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/freedos-user



___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[E. Auer]

Hi! Please describe that "normalizing" process which makes
the source codes inside the distro packages behave like a
solid archive. If you remove them from the ZIP and add a
TAR of the sources to the ZIP instead, you would get that
type of result, but it would mean that the install process
will have to be modified significantly. Also, you would
need temp storage for the TAR because DOS has fake pipes.


  2,586,758,742 bytes, original total ZIP size
  2,535,458,056 bytes, after `advzip -k -p -z -3 `


...


But normalizing the source code that is bundled in distro packages is
where the big savings happen because it makes the most compressible 
part of

the archive behave like a solid 7Z file.


...


The advzip output is indeed compatible with both pkunzip-2.04g and
pkunzip-2.50 running on a DOS machine with one megabyte of memory.


The way AdvanceCOMP checks and retains metadata is satisfying.  If you 
run
advzip on the entire FreeDOS collection, then you'll notice a 
non-trivial

number of malformed and/or corrupt ZIP files.


You mean they were already malformed before you recompressed them?
Can you provide a list of those damaged ZIP files to us?

Thanks! Regards, Eric




___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik,

> On Nov 11, 2021, at 1:56 PM, Darik Horn  wrote:
> [..]
> RAR implements the PKZIP listfile syntax, so the SETUP.BAT invocation
> would change to this;
> 
>  SET TCPU=8086
>  REM vinfo stuff goes here...
>  unrar x freedos.rar @%TCPU%.txt
> 
> Where the @LISTFILE corresponds to a SLICER /G %TCPU% build command.
> 
> Or excluding things that don't run on an 8086 or 80286 processor could
> be more concise:
> 
>  unrar x freedos.rar -x@%TCPU%.txt

Interesting. I assume it is capable of filtering multiple lists. 

>> requiring additional overhead for the install program and multiple passes 
>> through the spanned disks.
> 
> RAR does only one media pass.
> 
> (This is better than PKZIP, which reads the last floppy disk twice.)

Good. 

>> Honestly, I don’t really care about SLICER. Outside of the installer, it has 
>> almost no use and zero chance for any form of adoption by the community. I’d 
>> rather file it in the trash can, use an existing tool and work on other 
>> things. But to my knowledge, nothing else can do what it actually does in an 
>> acceptable manner.
> 
> The demo that I published can already do it.

If you can figure out how to get it to work, then it will be worth considering. 

It would come down a couple additional things. 

1) Wether or not it’s License is using an acceptable open source 
license. 
2) Usage is approved. Not up to me. But if licenses are ok, it should 
not be a problem.
3) Can be told to create spans of a specific size (so can be used on 
all floppy variations, one size fits all). 
4) Can replace SLICER with little to no overhead added to the installer 
and/or RBE (Release Build Environment)
5) Can provide the same level of flexibility without the need to 
manually modify code when there are package/tag changes.

These are of much less important and probable not “deal breakers". But, they 
are things slicer does do and should be known/considered. Can it display 
general text during the install process? For example, while installing from 
slow floppy diskettes, the user is given text regarding the history of FreeDOS 
to read at intervals for something to do while waiting. Does it support NLS and 
provide text in the users language for prompts?  On systems that support 
change-line, can it auto-resume without being told to continue? After 
completion, can it just return an errorlevel to the installer and not stop with 
an “I’m done, Press OK” message? 

> What are your SLICER /G inputs when building the installation media?
> -- I will redo the demo to more perfectly replicate current behavior.

Those and other settings are available in the projects SETTINGS directory. 
Specifically, the X86_ALL.LST defines what gets what slicer tags for assembling 
the archive. The lines label “# tags:” are NOT comments but are directives 
processed by the RBE to build the slicer archive. Oh, and that $PKG_ASSIST$ is 
a macro. The RBE will insert custom user assistance packages (like screen 
readers) at that point when it is being used to create customized versions of 
FreeDOS.

Obviously, the tags are cumulative and some things are just assumed for now. 
But, eventually they can be fine tuned and tags added/removed as needed. 

 https://github.com/shidel/FDI-x86/tree/master/SETTINGS 
 

Jerome

___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Darik Horn]

Jerome,

> I’m not sure what version of UnRAR you are using.

I used these two upstream releases to repack the latest FreeDOS 1.3
release candidate:

1. ftp://ftp.rarlab.com/rar/rar250.exe for decompression.

FreeDOS ships the 16-bit UnRAR 2.50 binary from this release, and the
IDOS.SFX module inside is what provides scripting features.

2. ftp://ftp.rarlab.com/rar/wrar290.exe for compression.

The console RAR 2.90 binary from this release has the LFN support
needed for FreeDOS source packages.


> I’m not sure what you mean by installation scripting.

Run rar250.exe for a demonstration.  It can make the FreeDOS installer
look and behave like the MS-DOS installer.
(Screenshot attached.)


> Lets take a couple different install situations as examples.
> ...
> Situation 1, Installation onto basic 286 w/EGA ...
> Situation 2, Install for 486 w/VGA ...
> Situation 3, Install VirtualBox, (i686 compatible) ...

RAR implements the PKZIP listfile syntax, so the SETUP.BAT invocation
would change to this;

  SET TCPU=8086
  REM vinfo stuff goes here...
  unrar x freedos.rar @%TCPU%.txt

Where the @LISTFILE corresponds to a SLICER /G %TCPU% build command.

Or excluding things that don't run on an 8086 or 80286 processor could
be more concise:

  unrar x freedos.rar -x@%TCPU%.txt


>  requiring additional overhead for the install program and multiple passes 
> through the spanned disks.

RAR does only one media pass.

(This is better than PKZIP, which reads the last floppy disk twice.)


> Honestly, I don’t really care about SLICER. Outside of the installer, it has 
> almost no use and zero chance for any form of adoption by the community. I’d 
> rather file it in the trash can, use an existing tool and work on other 
> things. But to my knowledge, nothing else can do what it actually does in an 
> acceptable manner.

The demo that I published can already do it.

What are your SLICER /G inputs when building the installation media?
-- I will redo the demo to more perfectly replicate current behavior.
___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Rugxulo]

Hi,

On Thu, Nov 11, 2021 at 5:05 AM Jerome Shidel  wrote:
>
> On Nov 11, 2021, at 12:12 AM, Darik Horn  wrote:
>
> The UnRAR in FreeDOS is 32,086 bytes and already implements all of the things 
> that you want for SCLICER, which is currently 28,188 bytes.
> (Compression, installation scripting, media spanning, and 160K compatibility.)
>
>
> I’m not sure what version of UnRAR you are using. But downloading the current 
> version that we have (there maybe and probably is a newer one somewhere)
> from 
> https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/repositories/latest/pkg-html/unrar.html
>  , it is 173,588 bytes. It is also listed as "Freeware, see license."

I never really used RAR. The resident expert around here would be
Laaca (Blocek). Although my brother registered WinRAR many years ago,
so I could maybe (barely) ask him for advice. It's a cool archiver,
but I've mostly only used ZIP and 7-Zip in recent years. (Newer
versions of RAR archives [v5?] won't unpack in DOS anymore.)

That old (DJGPP 2.04) build of UnRAR 3.93 was from me, so of course
it's bigger. I think he means even older RAR v2 (1999?), which did
have 16-bit DOS support. Unlike newer versions, that old decompression
code (unrarlib v2) has been GPL'd.

* https://www.sac.sk/download/pack/rar250.exe
* https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/util/file/unrar/v2/

AFAIK, last official DOS shareware release of RAR was RARX (EMX,
32-bit) from 2010:

* https://www.sac.sk/download/pack/rarx393.exe


___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Travis,

> On Nov 11, 2021, at 10:37 AM, Travis Siegel  wrote:
> 
> I have never used slicer, so can't comment on it. 
> 
I wouldn’t expect you to use it. In almost every case, there are other 
utilities that are in common use. And/or, better suited to a task.

Slicer was created specifically to make the Floppy Edition happen. Lets not 
forget that prior to 1.3-RC3, there was no Floppy Edition. I decided it was 
better to have the floppy version without compression. As opposed to waiting 
until compression was fully supported.
> However, I have used rar for years, and I've never found a task it can't be 
> made to do with proper command line parameters.  If you only want a specific 
> file extracted, it's easy enough to tell it to use that file only, (or a 
> range of files if that's your want). 
> 
Like I said in in earlier post, other utilities could be used to sort-of mimic 
what slicer does. But, it then it adds a lot of complexity to the installer and 
result in multiple passes through the archive. Or, at minimum require having 
the installer keep numerous lists and try to splice them together to feed the 
extractor. 

Off the top of my head, your looking at some unknown combination of 
8086,186,286,386,486,586,686 + Real,GenericVM,DOSBox,VirtualBox,VMware + 
MCGA,CGA,EGA,VGA + Network/not + UserLanguage

At present, some tags make no difference. While others do. And others (like 
maybe VESA) may be added at a later date.

How can that be organizes in a structure that does not require the installer to 
calculate lists, does not duplicate files and does not require multiple passes 
(diskette 1, diskette 2, diskette 1 again, ….)? 
> It can also extract with or without file paths.  It's been a while since I 
> compiled my own copy of unrar, but I honestly don't remember it being all 
> that large, it was certainly considerably smaller than the rar program 
> itself.  I don't (currently) have a working stand-alond dos machine, but I 
> think dosemu can help.  I'll do some digging/testing, and see if I can help 
> here, it's always better to have smaller distribution media, shorter download 
> timesequates to less time/money spent installing, and sometimes, that makes 
> all the difference as to whether someone goes with your solution, or chooses 
> something else.  If anything, I've always felt freedos doesn't include enough 
> in the way of programs for folks to get started, so if we can reduce the size 
> of the distribution media, then that will allow for including more programs 
> as well, and that should make everyone happy.
> 
I agree that smaller would be better. And assuming slicer is not replaced with 
another method that can do the same job, support for compression will come to 
it eventually. It is not a very complicated thing to add. It would probably 
only require a couple dozen lines of code. However, testing will be much more 
time consuming. Unfortunately, the list of other things to-do is long and many 
are of much higher importance.

:-)

Jerome


___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Travis Siegel]

I have never used slicer, so can't comment on it.  However, I have used 
rar for years, and I've never found a task it can't be made to do with 
proper command line parameters.  If you only want a specific file 
extracted, it's easy enough to tell it to use that file only, (or a 
range of files if that's your want).  It can also extract with or 
without file paths.  It's been a while since I compiled my own copy of 
unrar, but I honestly don't remember it being all that large, it was 
certainly considerably smaller than the rar program itself.  I don't 
(currently) have a working stand-alond dos machine, but I think dosemu 
can help.  I'll do some digging/testing, and see if I can help here, 
it's always better to have smaller distribution media, shorter download 
timesequates to less time/money spent installing, and sometimes, that 
makes all the difference as to whether someone goes with your solution, 
or chooses something else.  If anything, I've always felt freedos 
doesn't include enough in the way of programs for folks to get started, 
so if we can reduce the size of the distribution media, then that will 
allow for including more programs as well, and that should make everyone 
happy.



On 11/11/2021 6:04 AM, Jerome Shidel wrote:

Hi Darik,


On Nov 11, 2021, at 12:12 AM, Darik Horn  wrote:
[..]

SLICER is small. Less than 1/10th the size of RAR. While, this
does not matter as much for a 1.44mb diskette. It matters a lot
if a user only has a 160K drive to work with. With RAR being
roughly 353K, that is not going to happen.


The UnRAR in FreeDOS is 32,086 bytes and already implements all of 
the things that you want for SCLICER, which is currently 28,188 
bytes. (Compression, installation scripting, media spanning, and 160K 
compatibility.)


I’m not sure what version of UnRAR you are using. But downloading the 
current version that we have (there maybe and probably is a newer one 
somewhere) from 
https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/repositories/latest/pkg-html/unrar.html , 
it is 173,588 bytes. It is also listed as "Freeware, see license."


RAR does compression which is very nice. But, isn’t mandatory. It also 
does media spanning that is an absolute must have. I’m not sure what 
you mean by installation scripting. However, SLICER and RAR are very 
different when it comes to intended use. And to my knowledge RAR does 
not do all the things I need/want for the installer.


Lets take a couple different install situations as examples. (I’m 
going to paraphrase the command lines)


Situation 1, Installation onto basic 286 w/EGA

slicer extract 286,EGA programs to C:\

 or
unrar extract 8086\CGA\ directory to C:\
unrar extract 8086\EGA\ directory to C:\
unrar extract 186\CGA\ directory to C:\
unrar extract 186\EGA\ directory to C:\
unrar extract 286\CGA\ directory to C:\
unrar extract 286\EGA\ directory to C:\

Situation 2, Install for 486 w/VGA

slicer extract 486,VGA programs to C:\

 or
unrar extract 8086\CGA\ directory to C:\
unrar extract 8086\EGA\ directory to C:\
unrar extract 8086\VGA\ directory to C:\
unrar extract 186\CGA\ directory to C:\
unrar extract 186\EGA\ directory to C:\
unrar extract 186\VGA\ directory to C:\
unrar extract 286\CGA\ directory to C:\
unrar extract 286\EGA\ directory to C:\
unrar extract 286\VGA\ directory to C:\
unrar extract 386\CGA\ directory to C:\
unrar extract 386\EGA\ directory to C:\
unrar extract 386\VGA\ directory to C:\
unrar extract 486\CGA\ directory to C:\
unrar extract 486\EGA\ directory to C:\
unrar extract 486\VGA\ directory to C:\

Situation 3, Install VirtualBox, (i686 compatible)

slicer extract 686,VGA,VBOX,NETWORK programs to C:\

or

unrar …

Obviously, some of that could be consolidated and streamlined. There 
is a wide range of possible combinations. Hopefully, it demonstrates 
that there are differences between the two programs.


I haven’t really used RAR much in decades and I’m no expert on what 
can and cannot be done with it. Perhaps it could do something similar 
within the limits of a DOS command line without requiring additional 
overhead for the install program and multiple passes through the 
spanned disks.


Honestly, I don’t really care about SLICER. Outside of the installer, 
it has almost no use and zero chance for any form of adoption by the 
community. I’d rather file it in the trash can, use an existing tool 
and work on other things. But to my knowledge, nothing else can do 
what it actually does in an acceptable manner.


Eventually, SLICER will get support for compression. That won’t add 
much to the executable. It will not do compression itself but rely on 
other tools to perform those actions. Similar to TAR (which is 
actually a better comparison to SLICER), it will end up with support 
for a variety of external compression algorithms. That would provide 
more flexibility and vastly reduced archive size.


Compression support in SLICER won’t be in RC5. Maybe in 1.3-FINAL. 
But, 

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik,

> On Nov 11, 2021, at 12:12 AM, Darik Horn  wrote:
> [..]
> SLICER is small. Less than 1/10th the size of RAR. While, this does not 
> matter as much for a 1.44mb diskette. It matters a lot if a user only has a 
> 160K drive to work with. With RAR being roughly 353K, that is not going to 
> happen.
> 
> The UnRAR in FreeDOS is 32,086 bytes and already implements all of the things 
> that you want for SCLICER, which is currently 28,188 bytes. (Compression, 
> installation scripting, media spanning, and 160K compatibility.)

I’m not sure what version of UnRAR you are using. But downloading the current 
version that we have (there maybe and probably is a newer one somewhere) from 
https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/repositories/latest/pkg-html/unrar.html
 

 , it is 173,588 bytes. It is also listed as "Freeware, see license."

RAR does compression which is very nice. But, isn’t mandatory. It also does 
media spanning that is an absolute must have. I’m not sure what you mean by 
installation scripting. However, SLICER and RAR are very different when it 
comes to intended use. And to my knowledge RAR does not do all the things I 
need/want for the installer.

Lets take a couple different install situations as examples. (I’m going to 
paraphrase the command lines)

Situation 1, Installation onto basic 286 w/EGA

slicer extract 286,EGA programs to C:\

 or
unrar extract 8086\CGA\ directory to C:\ 
unrar extract 8086\EGA\ directory to C:\ 
unrar extract 186\CGA\ directory to C:\ 
unrar extract 186\EGA\ directory to C:\ 
unrar extract 286\CGA\ directory to C:\ 
unrar extract 286\EGA\ directory to C:\ 

Situation 2, Install for 486 w/VGA

slicer extract 486,VGA programs to C:\

 or
unrar extract 8086\CGA\ directory to C:\ 
unrar extract 8086\EGA\ directory to C:\ 
unrar extract 8086\VGA\ directory to C:\ 
unrar extract 186\CGA\ directory to C:\ 
unrar extract 186\EGA\ directory to C:\ 
unrar extract 186\VGA\ directory to C:\ 
unrar extract 286\CGA\ directory to C:\ 
unrar extract 286\EGA\ directory to C:\ 
unrar extract 286\VGA\ directory to C:\ 
unrar extract 386\CGA\ directory to C:\ 
unrar extract 386\EGA\ directory to C:\ 
unrar extract 386\VGA\ directory to C:\ 
unrar extract 486\CGA\ directory to C:\ 
unrar extract 486\EGA\ directory to C:\ 
unrar extract 486\VGA\ directory to C:\ 

Situation 3, Install VirtualBox, (i686 compatible)

slicer extract 686,VGA,VBOX,NETWORK programs to C:\

or

unrar …

Obviously, some of that could be consolidated and streamlined. There is a wide 
range of possible combinations. Hopefully, it demonstrates that there are 
differences between the two programs.  

I haven’t really used RAR much in decades and I’m no expert on what can and 
cannot be done with it. Perhaps it could do something similar within the limits 
of a DOS command line without requiring additional overhead for the install 
program and multiple passes through the spanned disks. 

Honestly, I don’t really care about SLICER. Outside of the installer, it has 
almost no use and zero chance for any form of adoption by the community. I’d 
rather file it in the trash can, use an existing tool and work on other things. 
But to my knowledge, nothing else can do what it actually does in an acceptable 
manner. 

Eventually, SLICER will get support for compression. That won’t add much to the 
executable. It will not do compression itself but rely on other tools to 
perform those actions. Similar to TAR (which is actually a better comparison to 
SLICER), it will end up with support for a variety of external compression 
algorithms. That would provide more flexibility and vastly reduced archive 
size. 

Compression support in SLICER won’t be in RC5. Maybe in 1.3-FINAL. But, I would 
count on it. 

> The corresponding RAR2 codec in unrarlib is GPL'd and approximately 12,000 
> bytes.
> 
> I haven't looked at the SLICER source code;

The current version isn’t pretty. It’s mostly a proof-of-concept rough draft. 
It should be rewritten from the ground up. But, I don’t really have the time 
for that either.

> this observation is more about how RAR is so good and how easy it was to 
> rebuild the FreeDOS installation media with it.

I get what your saying and agree that compression would be really really nice. 

:-)

Jerome


___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Darik Horn]

Eric,

I suggest a much shorter algorithm: Simply apply
> advzip with the "recompress" option to the ZIP
> files :-)


If you do this on freedos/files/repositories/latest today, then you'll get:

  2,586,758,742 bytes, original total ZIP size
  2,535,458,056 bytes, after `advzip -k -p -z -3 `

That's 51 MB, or 2%, for free.

But normalizing the source code that is bundled in distro packages is
where the big savings happen because it makes the most compressible part of
the archive behave like a solid 7Z file.


It is interesting that you also got savings with arj and bz2-zip,


ARJ has comparable performance, but BZIP2 is too slow to decompress on old
computers.

The zopfli method will get another few megabytes, but it needs more than a
day of CPU time to finish.


Of course before this is done on the whole repo,
> somebody with PKUNZIP2 has to volunteer to tell
> us whether an example advzipped ZIP is still fully
> compatible. As far as I remember, this was one of
> the design goals of advcomp, so I am optimistic.
>

The advzip output is indeed compatible with both pkunzip-2.04g and
pkunzip-2.50 running on a DOS machine with one megabyte of memory.

The way AdvanceCOMP checks and retains metadata is satisfying.  If you run
advzip on the entire FreeDOS collection, then you'll notice a non-trivial
number of malformed and/or corrupt ZIP files.
___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Darik Horn]

Jerome,

Recompression is something that will be considered for the next version of
> the repository management utilities.
>

Your insight is appreciated.  I saw the earlier discussion about media size
and wanted to point out that there are savings available now at nearly zero
cost.

SLICER is small. Less than 1/10th the size of RAR. While, this does not
> matter as much for a 1.44mb diskette. It matters a lot if a user only has a
> 160K drive to work with. With RAR being roughly 353K, that is not going to
> happen.
>

The UnRAR in FreeDOS is 32,086 bytes and already implements all of the
things that you want for SCLICER, which is currently 28,188 bytes.
(Compression, installation scripting, media spanning, and 160K
compatibility.)

The corresponding RAR2 codec in unrarlib is GPL'd and approximately 12,000
bytes.

I haven't looked at the SLICER source code; this observation is more about
how RAR is so good and how easy it was to rebuild the FreeDOS installation
media with it.
___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[E. Auer]

Hi Jerome and Darik,


  1. Unzip each FDN package.
  2. Further unpack each ZIP and 7Z source file.
  3. Zip+Store each SOURCES/* tree like it has LFN.
  4. Rezip each FDN package like usual.
  5. Optimize each package with AdvanceCOMP.


I suggest a much shorter algorithm: Simply apply
advzip with the "recompress" option to the ZIP
files :-) Without unpacking them and zipping them
again etc. Also, the advcomp/advmame tools can be
configured to put extreme effort in compression,
but from my experience with advpng, you already
get most of the possibe disk space reduction at
medium settings :-)

It is interesting that you also got savings with
arj and bz2-zip, but ZIP in the DOS compatible
variety (people with ancient PKZIP may enjoy the
ability to unzip without having to use INFOZIP)
I would not "overdo" things. Also, UNZIP can work
on old hardware with low RAM, which LZMA or BZIP
style algorithms do not necessarily offer. As far
as I remember, there may have been a license issue
behind not using RAR, but others may know more on
that point.

You can do mass-recompression of ZIP files with
advzip with your favorite shell (BASH for example)
so it does not take much effort to recompress the
whole repository once. I propose to use ZIP -o on
each file afterwards to reset the timestamps to
the newest timestamp of the contents, though!

Of course before this is done on the whole repo,
somebody with PKUNZIP2 has to volunteer to tell
us whether an example advzipped ZIP is still fully
compatible. As far as I remember, this was one of
the design goals of advcomp, so I am optimistic.

About the diskette edition: One question probably is
whether you want to use DOS from floppy or whether
you want to install it to harddisk. In the latter
case, you could just put everything into one tar or
7z with global (not per-file) compression applied
to have the smallest number of install disks, but
you would have less flexibility that way. And with
floppy, it is better when a broken sector only lets
installation of one of many zips fail instead of a
whole install.

However, I do not think that ANYBODY has drives
with less than 360k capacity. In particular, the
last time I touched a PC with 360k-only drives,
it did not have any harddisk, so you would not
need any type of installer to run there either.

Regards, Eric



___
Freedos-user mailing list
Freedos-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freedos-user

Re: [Freedos-user] ISO repack reduces size by 10%

[Jerome Shidel]

Hi Darik,

> On Nov 8, 2021, at 6:55 PM, Darik Horn  wrote:
> 
> Hi all,
> 
> The FreeDOS repo ISO file that was published today can be reduced from 632MB 
> to 579MB by repacking it. This adds 53MB of media headroom, which should 
> reduce pressure to bump packages in future point releases.

Just a note regarding the FreeDOS repo ISO... The Repo ISO image is mostly 
provided just as a convenience. The creation and publishing of that ISO is part 
of a fully automated process that is handled by the repository management 
utilities. Once per day in the early morning hours, the management utilities 
check to see if any packages have been added or updated. When it notices that 
occurred, it gathers all of the latest versions for each package and generates 
a ISO out of them. The creation of that ISO is not very complicated. However, 
it is not just a simple copy and burn image. It also goes through and generates 
listing and other metadata files for programs like FDNPKG and FDIMPLES. It also 
generates a bunch of “flat” html pages for that image as well.

There are several “packages” that are not in the repository at present that may 
be added later. There are various reasons for their current exclusion. 

For example, someday PC/GEOS[1] may be added. I have not discussed it with it’s 
developers for quite a long while. But, the last time I did, they did not feel 
it was ready for general public use/testing. That may be different now. Anyhow, 
the full version of that is around 120MB.  

I only see the total amount of software in the repository growing over time. 
Assuming we continue to provide a convenient download ISO that contains all of 
those packages, we will need a long term solution. That would probably involve 
splitting package groups onto separate images or possible a dvd iso. 

> 
> Original (as of 2021-11-08):
> https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/repositories/latest/cdrom.iso
>  
> 
> 
> Repack:
> https://drive.google.com/drive/folders/1b7IE-EsK5R1ROmXTaEvYNi9-kfVvH5wi?usp=sharing
>  
> 
> 
> The procedure is:
> 
>   1. Unzip each FDN package.
>   2. Further unpack each ZIP and 7Z source file.
>   3. Zip+Store each SOURCES/* tree like it has LFN.
>   4. Rezip each FDN package like usual.
>   5. Optimize each package with AdvanceCOMP.
> 
> The size at each step is:
> 
>   660,869,928 bytes, original
>   629,362,932 bytes, rezipped with InfoZip 3.0
>   605,766,821 bytes, optimized with AdvZip 2.1
> 
> Everything stays in zip format 2.0 and remains compatible with PKUnzip 2.04g 
> and FDNPKG.
> 
> Supposing that other ARCHIVER/* formats were implemented by the FreeDOS 
> packaging system, the cdrom.iso size could be:
> 
>   627,403,406 bytes, ARJ (-jm)
>   557,334,645 bytes, ZIP/BZ2 (-9)
>   495,380,042 bytes, RAR2 (-m5 -mdE -mm)
>   449,729,467 bytes, 7Z/LZMA2 (-mx9 -ms=on -mqs=on -md=1m)

The test results you’ve provided are interesting. They definitely show an 
improvement over the current packages. It would be better to recompress the 
packages when they are added to the repository. There are several benefits to 
doing it at that point.  As you have noted, it would reduce the space required 
on the repo ISO. It would also reduce the space used on the server and download 
size/time for individual packages. 

Recompression is something that will be considered for the next version of the 
repository management utilities. The current version of the repo utils is 
something like version 3 or 4. I haven’t really been counting. Nowadays, the 
management of the repo is fully automated. It makes things much easier to 
manage. For example, we can just push an updated package on the server and walk 
away.  It will do a bunch of verification stuff on the package and repository 
itself. Locate the older version. Version out the different copies. Adjust some 
symlinks for updater programs like FDNPKG. Create metadata and index files. 
Create/update static browsable HTML files from customizable templates. 
Maintains an RSS feed. Builds repository status vs previous OS release 
comparison charts. Publishes a package ISO.  And a couple other things. 

There are plans for a new version of the repo management utilities. There are a 
number of new features and things that can be improved. Just for example… Image 
and screen snapshots for packages. Improved directory versioning hierarchy. 
External software metadata (HTML pages for software not in the repo and link to 
an external website). And many many more.

> 
> UnRAR 2.50 is particularly impressive because it is 16-bit, fast, and will 
> extract on an 8088 with 640KB of memory. It also has an output size 
> competitive with the kind of 7-Zip 19 archive that can be extracted by 7ZDEC.
> 
> For fun, I also replaced SLICER with RAR and reduced the number of 
>