Thanks for all the tips, everyone! I had no idea VirtualT was scriptable via telnet. That is so convenient that I almost didn't bother writing my own tokenizer. But I realized that, now that I have a table of the tokens <http://fileformats.archiveteam.org/wiki/Tandy_200_BASIC_tokenized_file>, I could just plug it into a lexer like Flex and have it generate a C program for me. So that's what I did: https://github.com/hackerb9/tokenize. It should work for Model 100/102 and Tandy 200. Creating a version for the PC-8201/8300 could be as simple as plugging in the different token values <http://www.club100.org/library/ups/tokens.do>. I can't guarantee that it works for everything, but I tried it on a couple programs and it is giving me exactly the same file. (Okay, not "exactly" as I'm not calculating the memory offsets which the Model 100 recomputes when the file is loaded.)
Now I have another question: Are there different filename extensions for the different tokenized BASICs? I notice that the ROM2/Cleuseau documentation mentions that Tandy's BASIC and NEC's BASIC are tokenized differently. It has a "Strong machine file type mode" that forces the filename extension on a TDD to be *.BA1* for Model 100/102, *.BA2* for Tandy 200, and *.BA0* for PC-8201A. Are these well known enough extensions that I should use them in a new project? Or should I come up with something different, like *.BA.NEC.* —b9 On Mon, Jul 25, 2022 at 4:52 AM Ken Pettit <[email protected]> wrote: > M100/M102 and T200 all use the same token format, but 8201A/8300 is > different from the other 3 variants. You could try using M100 VirtualT > with 32K to tokenize and save the .BA file, then load it into T200. > > Ken > > On 7/24/22 11:42 PM, John R. Hogerhuis wrote: > > Not as far as I know. I think Robert Pigford's program is it. > > PowerBASIC is a compiler though so I guess its binaries would run > > under a Linux DOS emulator. I've certainly run DOS assemblers like > > TASM as part of a GNU/Linux Makefile. > > > > You would need a tokenizer for each BASIC variant you want to support > > (M100/102, 8201A/8300, and maybe t200?) > > > > The ROM of each machine is of course already capable of doing it > > properly by definition via LOAD/CLOAD. So emulators effectively > > support tokenization as well. VirtualT is fully scriptable over > > telnet/socket. > > > > For myself I just use my laptop or emulator to created tokenized files. > > > > One note... there are valid, specially constructed BA files that do > > not fully LIST, and are not convertible to ASCII BASIC text and back > > again, with high numbered lines and/or machine language embedded in > > it. To create such files goes beyond simple tokenization, but it may > > be useful and interesting to create advanced special BA binary file > > generators. Most such files require special measures to create. > > > > -- John. > >
