I've gotten co2ba.sh about as good as I think I'm going to get it for now.
It generates a larger block of loader code, which also runs slower, but
the file is somewhat smaller and that ends up making the total job take
about the same time with a 3.6k sample input co file.
The sample file used for these tests and comparisons is
ALTERN.CO manually reconstituted from
https://github.com/LivingM100SIG/Living_M100SIG/blob/main/M100SIG/Lib-07-UTILITIES/ALTERN.100
My previous loader code looks like this:
(extra blank lines to help reading after email wraps the long lines):
-----------
0CLEAR0,59346:A=59346:S=0:N$="ALTERN.CO":CLS:?"Installing "N$" ..";
1D$="":READD$:FORI=1TOLEN(D$)STEP2:B=(ASC(MID$(D$,I,1))-97)*16+ASC(MID$(D$,I+1,1))-97:POKEA,B:A=A+1:S=S+B:NEXT:?".";:IFA<62960THEN1
2IFS<>454932THEN?"Bad Checksum":END
3CALL59346
4DATAmndmpfmndbeccklppfolcbim...
-----------
With the new scheme I'm down to this
-----------
0READT:CLEAR2,T:DEFINTI,O,C,V,L:DEFSNGA,K,S,T,X,E:DEFSTRB,M,D,N:READT,L,X,K,N,O,M:E=T+L-1:A=T:S=0:C=0:CLS:PRINT"Installing
"N
1PRINT@20,CINT((L-(E-A))/L*100)"%":READD:FORI=1TOLEN(D):B=MID$(D,I,1):IFB=MTHENC=O:NEXT
2V=ASC(B)-C:POKEA,V:C=0:A=A+1:S=S+V:NEXT:IFA<=ETHEN1
3PRINT:IFS<>KTHENPRINT"Bad Checksum":END
4PRINT"Done. Please type: NEW":SAVEMN,T,E,X
5DATA59346,3614,59346,454932,"ALTERN.CO",64,"!"
6DATA"Í<õÍ1B*¿õë!a!DÍõ|µÊêç...
-----------
Part of the size difference is some apples/oranges differences that make
it not a direct comparison. The two could be more similar than this if I
wanted. Previously I just had the generator write the co header
variables directly in the code instead of having a header data line,
while in the new one I'm doing it all from a data line, because I like
that the loader code then is self contained & portable. You could copy
the loader block and stick it on top of some other paylod and it would work.
And another part is I made a real percent-done display on the new one
because it doesn't cost any run time, just a few more bytes of file
size. It only runs once per data line and outside of the inner loop.
The defint/defsng etc making line 0 longer also makes it run several
seconds faster.
I actually have an even slightly shorter version just by using the range
syntax for the DEF*
DEFINTA-E:DEFSNGF-K:DEFSTRL-O
vs
DEFINTI,O,C,V,L:DEFSNGA,K,S,T,X,E:DEFSTRB,M,D,N
but it makes the code just about unreadable since the letters lose all
meaning.
The notable points:
no goto in the inner loop, just next.
saved a line and also made it so that the generator script doesn't have
any forward references, so it can just increment line numbers without
having to hard code like a GOTO3 on line 1 etc.
Instead of
O=64 C=0 ... IFB=MTHENC=1 ... V=ASC(B)-(O*C)
(on every byte set a decode flag to 0 or 1, then multiply the encoding
offset by the on/off flag to enable/disable the offset)
Just
O=64 C=0 ... IFB=MTHENC=O ... V=ASC(B)-C
(instead of setting the encode flag to 0 or 1, just set it to 0 or the
actual offset value, then just subtract it directly without the
multiplication step. Always subtract, sometimes it's 0, sometimes its 64.
As far as I can tell, 0, 1, and 64 are all the same int and the same
work to process as long as the variables are declared to the same type.
Already mentioned all variables from data, can-nable loader code etc.
If the top address is the very first data value, you can read it, use it
to clear, and then just read it again to still have it after the clear
without wasting much space or cpu, and without needing the generator
script to write the value twice in duplicate assignments before & after
the clear.
Already mentioned the fancy percent-done progress.
The generator script has config options so you can change the behavior
at run-time by env variables.
So you can change the starting line number, the line number increment,
the length of the data lines, the encoding mark character, the encoding
offset value.
I have the generator script now counting all the bytes in the output
line when building data lines and deciding when to start a new line, so
now every line fills to the specified max length as much as possible
even though the size of the data varies because of the varied encoding.
All in all, the new way generates a smaller file, but the loader code
runs slower, and it ends up taking almost exactly the same total time to
load. The smaller file size is a win though, and the total time is
actually *slightly* in favor of the new way.
The new scheme is conceptually simple but it takes 2 lines of code and
includes an IF branch where the old way the entire loop is on a single
line and the the same math ops happen for every byte, no branching.
I read that one optimization is to move initialization/setup code to the
end instead of the top, and use goto or gosub to jump to it and back,
and have your tight loop as close to the top as possible. Something
about BASIC searching from the start of the file repeatedly? Well I
tried that and it made no difference in my run times. I tried both goto
and gosub.
For now I kept the old script in the repo as co2ba_old.sh since it's
output is probably still useful being all pure low ascii printable text.
Converting the same input:
ALTERN.CO 3620 bytes
old:
7749 bytes
xfer time: 1:05
load time: 2:04
total: 3:09
new:
5334 bytes
xfer time: 0:45
load time: 2:23
total: 3:07
https://github.com/bkw777/dl2/blob/master/co2ba.md
https://github.com/bkw777/dl2/blob/master/co2ba.sh
Anyway, thanks again for the idea Steve!
--
bkw
On 2/24/26 20:17, B 9 wrote:
Good points, Brian! I like the idea of using "!" so as to have fewer
special cases. I changed my co2do <https://github.com/hackerb9/co2do/>
script to match, kinda. I don’t quote spaces (32) or tab (9) which makes
the loader slightly smaller and that’s where the space matters.
One tricky thing is that Stephen started out the email thread by saying
±64, as you implemented, but then gave code showing ±128. I used the
latter approach because it tickles me that adding 128 and subtracting
128 are actually equivalent — and the same as xor — modulo 256. It also
suggests that there could be an efficient 8085 implementation, which may
be necessary for me (see below).
Using ±128 meant the EOF signal, ‘!\xFF’, was needed to encode Delete
(127), which I’m actually fine with. If there’s going to be an EOF
marker, I think it should be truly invalid (POKEing a negative number
would cause a |?FC Error|) instead of merely nonsensical. Like you, I
doubted the need for an EOF character, but left it in as I figured
Stephen probably had more experience than me on this.
By the way, I’m not sure if this will come back to bite me later, but
I’m also appending ^Z at the end of the .DO file, which makes it trivial
to send to the Model T without any transfer program.
|RUN "COM:88N1" cat FOO.DO >/dev/ttyUSB0|
Even though the encoding is fairly efficient, it could be better. Very
few control characters actually need to be quoted over the serial line:
they actually get lost in tokenization. But the biggest issue is that
the data in the BASIC program takes up RAM, potentially making it
impossible to CLEAR enough space for the .CO. That means that many .CO
files which could run fine on an 8K machine, will fail to load. I wonder
if there’s a way to make a less RAM hungry loader which is as simple to
use as the RUN/cat example above.
For instance, instead of having a large BASIC program in memory with the
entire .CO file encoded in it, perhaps it could be two stages, with the
first being a minimal program that receives the .CO file over the serial
port and POKEs it directly into the correct location. Essentially, what
I want is for |RUNM "COM:98N1"| to actually work, with the one addition
that it would call CLEAR first based on the .CO header.
I don’t know how to do this yet or if it is even possible. Perhaps, if
the .DO file had an embedded ^Z so that the BASIC program could start
executing and reading the rest of the data…. I know, I know. It’d be too
slow. Just brainstorming. Maybe if I could pop into machine language
quickly enough... Hrm. I may have to give up on the idea of making it as
convenient as RUN/cat.
Any suggestions/solutions are welcome!
—b9
On Tue, Feb 24, 2026 at 2:59 PM Brian K. White <[email protected]
<mailto:[email protected]>> wrote:
On 2/18/26 17:21, B 9 wrote:
> Uses Stephen's encoding with some tweaks. (Main difference is
that the
> end of data flag is changed to be an invalid sequence, "//", so
that DEL
> can be encoded).
I had the same initial thought about /255 as I was converting co2ba.sh
to use Steve's method.
"/ÿ" (/255) does not mean that you can't have an actual 255 in the
payload.
First, a /255 in the encoded data doesn't conflict with anything in the
payload because the payload only ever has /0+64 to /34+64 and /47+64
(/@
to /b and /o).
No other byte values will have a / in the first place.
Also, the decoder always subtracts 64 from the value to undo the fact
that the encoder added it to turn a control byte into a safe byte.
So even if a /255 were meant to be an encoded byte of payload, /ÿ would
decode to 255-64=191, but real 191 bytes are not encoded in the first
place. Nor are real 255 bytes.
So it's no risk or conflict to treat /255 (or /anything other than
0-31,34,47) for some other special purpose like end-of-data.
In my case, I'm using ! (33) as the escape char instead of / (47), and
just converting everything from 0-34 instead of adding a special cases
for " and /.
If I want the least encoder code, say for running on the 100 in the
smallest possible routine, I can just encode everything from 0-34 and
accept that 32's are encoded when they wouldn't need to be. Or if I
want
the smallest output I can add that one extra step in the encoder to
exclude 32.
It simplifies both the encoder and decoder slightly.
Also in my case I'm dispensing with the /255 eof mechanism altogether
because we always know the length and can just as easily write the
length in the output as write an eof mark. Most of the time the data
will be a CO file which needs the length info in it's header anyway,
but
even for arbitrary binary data without a CO header, it's still both a
simpler and more robust loop to just count from start value to end
value
instead of count forever and perform a check on every byte along the
way
to see if we got an eof, (and hope we actually do always get one!).
Similarly I also write a checksum along with the length.
--
bkw
--
bkw
