Sweet! It's surprising that the loader slowed down that much, but as you said, the small size is a win and it's still slightly faster if you count the transfer time, so good job! Do you have an estimate for how large of a .CO file your script can handle now?
By the way, even though you specified 9600 baud, you said it took 45 seconds to transfer 5334 bytes, which is less than 1200 baud. I tried your script on my T200 and the serial transfer took only 7 seconds. (I used `LOAD "COM:88N1ENN"` on the 5371 byte file co2ba.sh generated and measured with `time cat OUT.DO > /dev/ttyUSB0`) Tokenization on my T200 seems to limit the transfer speed to around 9600. Is the M100 tokenization significantly slower? If so, that suggests to me strongly that the solution may be a tiny loader that then receives the !-encoded data over the serial port. Someone (John Hogerhuis?) posted on the Bitchin100 Wiki <https://bitchin100.com/wiki/index.php?title=Model_100_Serial_Interface#Baud_Rate> some BASIC code that can transfer data at about 9600 baud and assembly code that can talk at 38,400. I like your idea of using data statements for showing percentage complete. Printing the byte count was slowing down the loader so much, I had been considering completely removing the progress display from my co2do. Ignoring the serial transfer, but taking the encoding scheme into account, it looks like your loader operates at about 200 baud (3620 bytes in 2:23) which is close to what my BASIC co2do loader was getting. I am unhappy with that speed as it is actually faster to send TEENY over, launch it, remember the arcane syntax, type in the L command, notice I forgot to start the dl server, do so, type in the command again, and receive the .CO file directly. So, I've been exploring the opposite direction: a more complex loader that runs faster. Something like TEENY or TBACK, but without a dependency on a special server on the PC side. I'll let you all know if I figure something out. John: Did you ever release the source code for TBACK? —b9 P.S. Just for future reference, using VARPTR and PEEK in BASIC is no faster at getting a single character from a string than using MID$(...). On Wed, Mar 4, 2026 at 2:10 PM Brian K. White <[email protected]> wrote: > > I've gotten co2ba.sh about as good as I think I'm going to get it for now. > > It generates a larger block of loader code, which also runs slower, but > the file is somewhat smaller and that ends up making the total job take > about the same time with a 3.6k sample input co file. > > > The sample file used for these tests and comparisons is > ALTERN.CO manually reconstituted from > > https://github.com/LivingM100SIG/Living_M100SIG/blob/main/M100SIG/Lib-07-UTILITIES/ALTERN.100 > > > > My previous loader code looks like this: > (extra blank lines to help reading after email wraps the long lines): > > ----------- > 0CLEAR0,59346:A=59346:S=0:N$="ALTERN.CO":CLS:?"Installing "N$" .."; > > > 1D$="":READD$:FORI=1TOLEN(D$)STEP2:B=(ASC(MID$(D$,I,1))-97)*16+ASC(MID$(D$,I+1,1))-97:POKEA,B:A=A+1:S=S+B:NEXT:?".";:IFA<62960THEN1 > > 2IFS<>454932THEN?"Bad Checksum":END > > 3CALL59346 > > 4DATAmndmpfmndbeccklppfolcbim... > ----------- > > > > > > With the new scheme I'm down to this > > ----------- > 0READT:CLEAR2,T:DEFINTI,O,C,V,L:DEFSNGA,K,S,T,X,E:DEFSTRB,M,D,N:READT,L,X,K,N,O,M:E=T+L-1:A=T:S=0:C=0:CLS:PRINT"Installing > > "N > > > 1PRINT@20 > ,CINT((L-(E-A))/L*100)"%":READD:FORI=1TOLEN(D):B=MID$(D,I,1):IFB=MTHENC=O:NEXT > > > 2V=ASC(B)-C:POKEA,V:C=0:A=A+1:S=S+V:NEXT:IFA<=ETHEN1 > > > 3PRINT:IFS<>KTHENPRINT"Bad Checksum":END > > > 4PRINT"Done. Please type: NEW":SAVEMN,T,E,X > > > 5DATA59346,3614,59346,454932,"ALTERN.CO",64,"!" > > 6DATA"Í<õÍ1B*¿õë!aŒ!DÍ õ|µÊêç... > ----------- > > > Part of the size difference is some apples/oranges differences that make > it not a direct comparison. The two could be more similar than this if I > wanted. Previously I just had the generator write the co header > variables directly in the code instead of having a header data line, > while in the new one I'm doing it all from a data line, because I like > that the loader code then is self contained & portable. You could copy > the loader block and stick it on top of some other paylod and it would > work. > > And another part is I made a real percent-done display on the new one > because it doesn't cost any run time, just a few more bytes of file > size. It only runs once per data line and outside of the inner loop. > > The defint/defsng etc making line 0 longer also makes it run several > seconds faster. > > I actually have an even slightly shorter version just by using the range > syntax for the DEF* > DEFINTA-E:DEFSNGF-K:DEFSTRL-O > vs > DEFINTI,O,C,V,L:DEFSNGA,K,S,T,X,E:DEFSTRB,M,D,N > but it makes the code just about unreadable since the letters lose all > meaning. > > The notable points: > > no goto in the inner loop, just next. > saved a line and also made it so that the generator script doesn't have > any forward references, so it can just increment line numbers without > having to hard code like a GOTO3 on line 1 etc. > > Instead of > O=64 C=0 ... IFB=MTHENC=1 ... V=ASC(B)-(O*C) > (on every byte set a decode flag to 0 or 1, then multiply the encoding > offset by the on/off flag to enable/disable the offset) > > Just > O=64 C=0 ... IFB=MTHENC=O ... V=ASC(B)-C > (instead of setting the encode flag to 0 or 1, just set it to 0 or the > actual offset value, then just subtract it directly without the > multiplication step. Always subtract, sometimes it's 0, sometimes its 64. > As far as I can tell, 0, 1, and 64 are all the same int and the same > work to process as long as the variables are declared to the same type. > > Already mentioned all variables from data, can-nable loader code etc. > > If the top address is the very first data value, you can read it, use it > to clear, and then just read it again to still have it after the clear > without wasting much space or cpu, and without needing the generator > script to write the value twice in duplicate assignments before & after > the clear. > > Already mentioned the fancy percent-done progress. > > The generator script has config options so you can change the behavior > at run-time by env variables. > > So you can change the starting line number, the line number increment, > the length of the data lines, the encoding mark character, the encoding > offset value. > > I have the generator script now counting all the bytes in the output > line when building data lines and deciding when to start a new line, so > now every line fills to the specified max length as much as possible > even though the size of the data varies because of the varied encoding. > > All in all, the new way generates a smaller file, but the loader code > runs slower, and it ends up taking almost exactly the same total time to > load. The smaller file size is a win though, and the total time is > actually *slightly* in favor of the new way. > > The new scheme is conceptually simple but it takes 2 lines of code and > includes an IF branch where the old way the entire loop is on a single > line and the the same math ops happen for every byte, no branching. > > I read that one optimization is to move initialization/setup code to the > end instead of the top, and use goto or gosub to jump to it and back, > and have your tight loop as close to the top as possible. Something > about BASIC searching from the start of the file repeatedly? Well I > tried that and it made no difference in my run times. I tried both goto > and gosub. > > For now I kept the old script in the repo as co2ba_old.sh since it's > output is probably still useful being all pure low ascii printable text. > > Converting the same input: > ALTERN.CO 3620 bytes > > old: > 7749 bytes > xfer time: 1:05 > load time: 2:04 > total: 3:09 > > new: > 5334 bytes > xfer time: 0:45 > load time: 2:23 > total: 3:07 > > https://github.com/bkw777/dl2/blob/master/co2ba.md > https://github.com/bkw777/dl2/blob/master/co2ba.sh > > > Anyway, thanks again for the idea Steve! > > -- > bkw > > On 2/24/26 20:17, B 9 wrote: > > Good points, Brian! I like the idea of using "!" so as to have fewer > > special cases. I changed my co2do <https://github.com/hackerb9/co2do/> > > script to match, kinda. I don’t quote spaces (32) or tab (9) which makes > > the loader slightly smaller and that’s where the space matters. > > > > One tricky thing is that Stephen started out the email thread by saying > > ±64, as you implemented, but then gave code showing ±128. I used the > > latter approach because it tickles me that adding 128 and subtracting > > 128 are actually equivalent — and the same as xor — modulo 256. It also > > suggests that there could be an efficient 8085 implementation, which may > > be necessary for me (see below). > > > > Using ±128 meant the EOF signal, ‘!\xFF’, was needed to encode Delete > > (127), which I’m actually fine with. If there’s going to be an EOF > > marker, I think it should be truly invalid (POKEing a negative number > > would cause a |?FC Error|) instead of merely nonsensical. Like you, I > > doubted the need for an EOF character, but left it in as I figured > > Stephen probably had more experience than me on this. > > > > By the way, I’m not sure if this will come back to bite me later, but > > I’m also appending ^Z at the end of the .DO file, which makes it trivial > > to send to the Model T without any transfer program. > > > > |RUN "COM:88N1" cat FOO.DO >/dev/ttyUSB0| > > > > Even though the encoding is fairly efficient, it could be better. Very > > few control characters actually need to be quoted over the serial line: > > they actually get lost in tokenization. But the biggest issue is that > > the data in the BASIC program takes up RAM, potentially making it > > impossible to CLEAR enough space for the .CO. That means that many .CO > > files which could run fine on an 8K machine, will fail to load. I wonder > > if there’s a way to make a less RAM hungry loader which is as simple to > > use as the RUN/cat example above. > > > > For instance, instead of having a large BASIC program in memory with the > > entire .CO file encoded in it, perhaps it could be two stages, with the > > first being a minimal program that receives the .CO file over the serial > > port and POKEs it directly into the correct location. Essentially, what > > I want is for |RUNM "COM:98N1"| to actually work, with the one addition > > that it would call CLEAR first based on the .CO header. > > > > I don’t know how to do this yet or if it is even possible. Perhaps, if > > the .DO file had an embedded ^Z so that the BASIC program could start > > executing and reading the rest of the data…. I know, I know. It’d be too > > slow. Just brainstorming. Maybe if I could pop into machine language > > quickly enough... Hrm. I may have to give up on the idea of making it as > > convenient as RUN/cat. > > > > Any suggestions/solutions are welcome! > > > > —b9 > > > > > > On Tue, Feb 24, 2026 at 2:59 PM Brian K. White <[email protected] > > <mailto:[email protected]>> wrote: > > > > On 2/18/26 17:21, B 9 wrote: > > > Uses Stephen's encoding with some tweaks. (Main difference is > > that the > > > end of data flag is changed to be an invalid sequence, "//", so > > that DEL > > > can be encoded). > > > > I had the same initial thought about /255 as I was converting > co2ba.sh > > to use Steve's method. > > > > "/ÿ" (/255) does not mean that you can't have an actual 255 in the > > payload. > > > > First, a /255 in the encoded data doesn't conflict with anything in > the > > payload because the payload only ever has /0+64 to /34+64 and /47+64 > > (/@ > > to /b and /o). > > > > No other byte values will have a / in the first place. > > > > Also, the decoder always subtracts 64 from the value to undo the fact > > that the encoder added it to turn a control byte into a safe byte. > > > > So even if a /255 were meant to be an encoded byte of payload, /ÿ > would > > decode to 255-64=191, but real 191 bytes are not encoded in the first > > place. Nor are real 255 bytes. > > > > So it's no risk or conflict to treat /255 (or /anything other than > > 0-31,34,47) for some other special purpose like end-of-data. > > > > > > In my case, I'm using ! (33) as the escape char instead of / (47), > and > > just converting everything from 0-34 instead of adding a special > cases > > for " and /. > > > > If I want the least encoder code, say for running on the 100 in the > > smallest possible routine, I can just encode everything from 0-34 and > > accept that 32's are encoded when they wouldn't need to be. Or if I > > want > > the smallest output I can add that one extra step in the encoder to > > exclude 32. > > > > It simplifies both the encoder and decoder slightly. > > > > Also in my case I'm dispensing with the /255 eof mechanism altogether > > because we always know the length and can just as easily write the > > length in the output as write an eof mark. Most of the time the data > > will be a CO file which needs the length info in it's header anyway, > > but > > even for arbitrary binary data without a CO header, it's still both a > > simpler and more robust loop to just count from start value to end > > value > > instead of count forever and perform a check on every byte along the > > way > > to see if we got an eof, (and hope we actually do always get one!). > > > > Similarly I also write a checksum along with the length. > > > > -- > > bkw > > > > > -- > bkw >
