Transfer time was limited by the default per-byte artificially added delay in dl -b, which has gotten larger every time I encounter some combination of a file and a receiving machine that fails.
The default delay is enough to handle the worst case I ever encountered, and I haven't had to increase it in a while so it might be done increasing. So for instance, for most files on most machines, 5ms is probably enough. I think that was the default in mComm. Then one day some particular basic code going to say an NEC always failed. The transfer would seem to go fine, but in the end there would be a missing byte somewhere and it would crash while running. The missing byte was consistent, always the same byte in the same place every time, but only with that particular file. The file itself is valid and runs fine if not corrupt. Another file with different but equivalent contents always worked fine. But also just increasing the delay to 6ms and no other changes also fixed it. So basic must be doing *some* level of parsing and work at that stage during transfer even though it's not executing the code yet, just storing it. Otherwise all files would behave like all other files. Over time I hit a few other magic bad combos and each time the delay increased until I stopped having failures no matter what data to what machine. I think it's at 8ms now. (on my phone not looking) The delay is configurable at run time with -z so I could say "dl -vb file.do -z 5" but I don't bother. I think the last example was one of the teeny or tsdos loaders for 200, and it would always fail to transfer a certain ? in the middle of a certain line, always the same spot, resulting in an invalid line. The ? was a print not part of a string. So the glitch was at a statement boundary like ...:?... It doesn't seem related to line length. That last example I think did appear after I modified an existing working loader to have fewer longer lines, but other even longer lines are no problem, and the glitch was in the middle of a line. I didn't realize all this time that simply having REX installed might have been taxing so much and affect serial transfers, so it's possible that was a factor. I have REX classic installed or not installed basically at random on any given machine at any given time. I should try that worst case example again and see if rex changes it. I did document the particulars somewhere so I can reproduce it, possibly in the dl2 source or the ref dir or just a file on my machine. Originally co2ba.sh was just a quick n dirty accessory to dl2 to be able to generate a loader for any random .co on the spot, and not to produce the best most crafted loaders possible, so the speed doesn't *really* matter so much, but the file size definitely does. So the new encoding is great. There are not a lot of .co files out there that don't already have loaders but I use it for rex classic setup to skip the step of installing and using teeny just to transfer one file and then immediately kill teeny to run that same file. I also used it for ramdsk for 100 & 200 for node datapac / rampac / minindp I am very interested in your machine language method. If it can be universal that would be huge. I have actually every machine to test on, even kc85. By "universal" it's good enough to have to know which kind of machine to target at generate-time. I already have to tell co2ba whether the output should say savem or bsave etc, rather than have code in the loader to figure that out. The binary being loaded is never portable anyway so there is not much point in a loader that would also work on the wrong machine for the payload binary. I just mean that it doesn't assume anything like special addresses. Running directly out of a string sure sounds like that. bkw On Fri, Mar 6, 2026, 1:19 AM B 9 <[email protected]> wrote: > Sweet! It's surprising that the loader slowed down that much, but as you > said, the small size is a win and it's still slightly faster if you count > the transfer time, so good job! Do you have an estimate for how large of a > .CO file your script can handle now? > > By the way, even though you specified 9600 baud, you said it took 45 > seconds to transfer 5334 bytes, which is less than 1200 baud. I tried your > script on my T200 and the serial transfer took only 7 seconds. (I used > `LOAD "COM:88N1ENN"` on the 5371 byte file co2ba.sh generated and measured > with `time cat OUT.DO > /dev/ttyUSB0`) > > Tokenization on my T200 seems to limit the transfer speed to around > 9600. Is the M100 tokenization significantly slower? If so, that suggests > to me strongly that the solution may be a tiny loader that then receives > the !-encoded data over the serial port. Someone (John Hogerhuis?) posted > on the Bitchin100 Wiki > <https://bitchin100.com/wiki/index.php?title=Model_100_Serial_Interface#Baud_Rate> > some BASIC code that can transfer data at about 9600 baud and assembly code > that can talk at 38,400. > > I like your idea of using data statements for showing percentage complete. > Printing the byte count was slowing down the loader so much, I had been > considering completely removing the progress display from my co2do. > > Ignoring the serial transfer, but taking the encoding scheme into account, > it looks like your loader operates at about 200 baud (3620 bytes in 2:23) > which is close to what my BASIC co2do loader was getting. I am unhappy with > that speed as it is actually faster to send TEENY over, launch it, remember > the arcane syntax, type in the L command, notice I forgot to start the dl > server, do so, type in the command again, and receive the .CO file > directly. > > So, I've been exploring the opposite direction: a more complex loader that > runs faster. Something like TEENY or TBACK, but without a dependency on a > special server on the PC side. I'll let you all know if I figure something > out. > > John: Did you ever release the source code for TBACK? > > —b9 > > P.S. Just for future reference, using VARPTR and PEEK in BASIC is no > faster at getting a single character from a string than using MID$(...). > > On Wed, Mar 4, 2026 at 2:10 PM Brian K. White <[email protected]> > wrote: > >> >> I've gotten co2ba.sh about as good as I think I'm going to get it for now. >> >> It generates a larger block of loader code, which also runs slower, but >> the file is somewhat smaller and that ends up making the total job take >> about the same time with a 3.6k sample input co file. >> >> >> The sample file used for these tests and comparisons is >> ALTERN.CO manually reconstituted from >> >> https://github.com/LivingM100SIG/Living_M100SIG/blob/main/M100SIG/Lib-07-UTILITIES/ALTERN.100 >> >> >> >> My previous loader code looks like this: >> (extra blank lines to help reading after email wraps the long lines): >> >> ----------- >> 0CLEAR0,59346:A=59346:S=0:N$="ALTERN.CO":CLS:?"Installing "N$" .."; >> >> >> 1D$="":READD$:FORI=1TOLEN(D$)STEP2:B=(ASC(MID$(D$,I,1))-97)*16+ASC(MID$(D$,I+1,1))-97:POKEA,B:A=A+1:S=S+B:NEXT:?".";:IFA<62960THEN1 >> >> 2IFS<>454932THEN?"Bad Checksum":END >> >> 3CALL59346 >> >> 4DATAmndmpfmndbeccklppfolcbim... >> ----------- >> >> >> >> >> >> With the new scheme I'm down to this >> >> ----------- >> 0READT:CLEAR2,T:DEFINTI,O,C,V,L:DEFSNGA,K,S,T,X,E:DEFSTRB,M,D,N:READT,L,X,K,N,O,M:E=T+L-1:A=T:S=0:C=0:CLS:PRINT"Installing >> >> "N >> >> >> 1PRINT@20 >> ,CINT((L-(E-A))/L*100)"%":READD:FORI=1TOLEN(D):B=MID$(D,I,1):IFB=MTHENC=O:NEXT >> >> >> 2V=ASC(B)-C:POKEA,V:C=0:A=A+1:S=S+V:NEXT:IFA<=ETHEN1 >> >> >> 3PRINT:IFS<>KTHENPRINT"Bad Checksum":END >> >> >> 4PRINT"Done. Please type: NEW":SAVEMN,T,E,X >> >> >> 5DATA59346,3614,59346,454932,"ALTERN.CO",64,"!" >> >> 6DATA"Í<õÍ1B*¿õë!aŒ!DÍ õ|µÊêç... >> ----------- >> >> >> Part of the size difference is some apples/oranges differences that make >> it not a direct comparison. The two could be more similar than this if I >> wanted. Previously I just had the generator write the co header >> variables directly in the code instead of having a header data line, >> while in the new one I'm doing it all from a data line, because I like >> that the loader code then is self contained & portable. You could copy >> the loader block and stick it on top of some other paylod and it would >> work. >> >> And another part is I made a real percent-done display on the new one >> because it doesn't cost any run time, just a few more bytes of file >> size. It only runs once per data line and outside of the inner loop. >> >> The defint/defsng etc making line 0 longer also makes it run several >> seconds faster. >> >> I actually have an even slightly shorter version just by using the range >> syntax for the DEF* >> DEFINTA-E:DEFSNGF-K:DEFSTRL-O >> vs >> DEFINTI,O,C,V,L:DEFSNGA,K,S,T,X,E:DEFSTRB,M,D,N >> but it makes the code just about unreadable since the letters lose all >> meaning. >> >> The notable points: >> >> no goto in the inner loop, just next. >> saved a line and also made it so that the generator script doesn't have >> any forward references, so it can just increment line numbers without >> having to hard code like a GOTO3 on line 1 etc. >> >> Instead of >> O=64 C=0 ... IFB=MTHENC=1 ... V=ASC(B)-(O*C) >> (on every byte set a decode flag to 0 or 1, then multiply the encoding >> offset by the on/off flag to enable/disable the offset) >> >> Just >> O=64 C=0 ... IFB=MTHENC=O ... V=ASC(B)-C >> (instead of setting the encode flag to 0 or 1, just set it to 0 or the >> actual offset value, then just subtract it directly without the >> multiplication step. Always subtract, sometimes it's 0, sometimes its 64. >> As far as I can tell, 0, 1, and 64 are all the same int and the same >> work to process as long as the variables are declared to the same type. >> >> Already mentioned all variables from data, can-nable loader code etc. >> >> If the top address is the very first data value, you can read it, use it >> to clear, and then just read it again to still have it after the clear >> without wasting much space or cpu, and without needing the generator >> script to write the value twice in duplicate assignments before & after >> the clear. >> >> Already mentioned the fancy percent-done progress. >> >> The generator script has config options so you can change the behavior >> at run-time by env variables. >> >> So you can change the starting line number, the line number increment, >> the length of the data lines, the encoding mark character, the encoding >> offset value. >> >> I have the generator script now counting all the bytes in the output >> line when building data lines and deciding when to start a new line, so >> now every line fills to the specified max length as much as possible >> even though the size of the data varies because of the varied encoding. >> >> All in all, the new way generates a smaller file, but the loader code >> runs slower, and it ends up taking almost exactly the same total time to >> load. The smaller file size is a win though, and the total time is >> actually *slightly* in favor of the new way. >> >> The new scheme is conceptually simple but it takes 2 lines of code and >> includes an IF branch where the old way the entire loop is on a single >> line and the the same math ops happen for every byte, no branching. >> >> I read that one optimization is to move initialization/setup code to the >> end instead of the top, and use goto or gosub to jump to it and back, >> and have your tight loop as close to the top as possible. Something >> about BASIC searching from the start of the file repeatedly? Well I >> tried that and it made no difference in my run times. I tried both goto >> and gosub. >> >> For now I kept the old script in the repo as co2ba_old.sh since it's >> output is probably still useful being all pure low ascii printable text. >> >> Converting the same input: >> ALTERN.CO 3620 bytes >> >> old: >> 7749 bytes >> xfer time: 1:05 >> load time: 2:04 >> total: 3:09 >> >> new: >> 5334 bytes >> xfer time: 0:45 >> load time: 2:23 >> total: 3:07 >> >> https://github.com/bkw777/dl2/blob/master/co2ba.md >> https://github.com/bkw777/dl2/blob/master/co2ba.sh >> >> >> Anyway, thanks again for the idea Steve! >> >> -- >> bkw >> >> On 2/24/26 20:17, B 9 wrote: >> > Good points, Brian! I like the idea of using "!" so as to have fewer >> > special cases. I changed my co2do <https://github.com/hackerb9/co2do/> >> > script to match, kinda. I don’t quote spaces (32) or tab (9) which >> makes >> > the loader slightly smaller and that’s where the space matters. >> > >> > One tricky thing is that Stephen started out the email thread by saying >> > ±64, as you implemented, but then gave code showing ±128. I used the >> > latter approach because it tickles me that adding 128 and subtracting >> > 128 are actually equivalent — and the same as xor — modulo 256. It also >> > suggests that there could be an efficient 8085 implementation, which >> may >> > be necessary for me (see below). >> > >> > Using ±128 meant the EOF signal, ‘!\xFF’, was needed to encode Delete >> > (127), which I’m actually fine with. If there’s going to be an EOF >> > marker, I think it should be truly invalid (POKEing a negative number >> > would cause a |?FC Error|) instead of merely nonsensical. Like you, I >> > doubted the need for an EOF character, but left it in as I figured >> > Stephen probably had more experience than me on this. >> > >> > By the way, I’m not sure if this will come back to bite me later, but >> > I’m also appending ^Z at the end of the .DO file, which makes it >> trivial >> > to send to the Model T without any transfer program. >> > >> > |RUN "COM:88N1" cat FOO.DO >/dev/ttyUSB0| >> > >> > Even though the encoding is fairly efficient, it could be better. Very >> > few control characters actually need to be quoted over the serial line: >> > they actually get lost in tokenization. But the biggest issue is that >> > the data in the BASIC program takes up RAM, potentially making it >> > impossible to CLEAR enough space for the .CO. That means that many .CO >> > files which could run fine on an 8K machine, will fail to load. I >> wonder >> > if there’s a way to make a less RAM hungry loader which is as simple to >> > use as the RUN/cat example above. >> > >> > For instance, instead of having a large BASIC program in memory with >> the >> > entire .CO file encoded in it, perhaps it could be two stages, with the >> > first being a minimal program that receives the .CO file over the >> serial >> > port and POKEs it directly into the correct location. Essentially, what >> > I want is for |RUNM "COM:98N1"| to actually work, with the one addition >> > that it would call CLEAR first based on the .CO header. >> > >> > I don’t know how to do this yet or if it is even possible. Perhaps, if >> > the .DO file had an embedded ^Z so that the BASIC program could start >> > executing and reading the rest of the data…. I know, I know. It’d be >> too >> > slow. Just brainstorming. Maybe if I could pop into machine language >> > quickly enough... Hrm. I may have to give up on the idea of making it >> as >> > convenient as RUN/cat. >> > >> > Any suggestions/solutions are welcome! >> > >> > —b9 >> > >> > >> > On Tue, Feb 24, 2026 at 2:59 PM Brian K. White <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > On 2/18/26 17:21, B 9 wrote: >> > > Uses Stephen's encoding with some tweaks. (Main difference is >> > that the >> > > end of data flag is changed to be an invalid sequence, "//", so >> > that DEL >> > > can be encoded). >> > >> > I had the same initial thought about /255 as I was converting >> co2ba.sh >> > to use Steve's method. >> > >> > "/ÿ" (/255) does not mean that you can't have an actual 255 in the >> > payload. >> > >> > First, a /255 in the encoded data doesn't conflict with anything in >> the >> > payload because the payload only ever has /0+64 to /34+64 and /47+64 >> > (/@ >> > to /b and /o). >> > >> > No other byte values will have a / in the first place. >> > >> > Also, the decoder always subtracts 64 from the value to undo the >> fact >> > that the encoder added it to turn a control byte into a safe byte. >> > >> > So even if a /255 were meant to be an encoded byte of payload, /ÿ >> would >> > decode to 255-64=191, but real 191 bytes are not encoded in the >> first >> > place. Nor are real 255 bytes. >> > >> > So it's no risk or conflict to treat /255 (or /anything other than >> > 0-31,34,47) for some other special purpose like end-of-data. >> > >> > >> > In my case, I'm using ! (33) as the escape char instead of / (47), >> and >> > just converting everything from 0-34 instead of adding a special >> cases >> > for " and /. >> > >> > If I want the least encoder code, say for running on the 100 in the >> > smallest possible routine, I can just encode everything from 0-34 >> and >> > accept that 32's are encoded when they wouldn't need to be. Or if I >> > want >> > the smallest output I can add that one extra step in the encoder to >> > exclude 32. >> > >> > It simplifies both the encoder and decoder slightly. >> > >> > Also in my case I'm dispensing with the /255 eof mechanism >> altogether >> > because we always know the length and can just as easily write the >> > length in the output as write an eof mark. Most of the time the data >> > will be a CO file which needs the length info in it's header anyway, >> > but >> > even for arbitrary binary data without a CO header, it's still both >> a >> > simpler and more robust loop to just count from start value to end >> > value >> > instead of count forever and perform a check on every byte along the >> > way >> > to see if we got an eof, (and hope we actually do always get one!). >> > >> > Similarly I also write a checksum along with the length. >> > >> > -- >> > bkw >> > >> >> >> -- >> bkw >> >
