Hi!
I think it's more readable if I combine all thoughts in a new message,
instead of replying to multiple messages. I'll try to separate different
design issues, to we can discuss each of them apart from the others.
1. Just ROMs vs all games
I think Ricardo has a good point when he says that ROM/DSK/BASIC/COM doesn't
make much sense to most emulator users. Since we're designing a new format
anyway, we might as well go all the way and create a format that can store
any MSX game.
2. Single file vs multiple files
Single file has the advantage of easy storage and distribution. Multiple
files allow a simpler format, keeping the game data separate from other files.
Archives combine the advantages: there is a single file on the outside, but
multiple files on the inside. The downside is that it's a bit more work to
manage, but there is code available (like ZLIB) so this isn't too much of a
problem.
2A. Filename limits
I only thought of this when this mail was almost finished and I didn't want
to renumber everything, that's why it's numbered 2A instead of 3.
Are the filenames limited to 8.3 (DOS) or can it be any length? What
characters is it allowed to contain (spaces, unicode)? Is it case sensitive?
In the case of an archive, these questions apply to the names of files inside
the archive, the filename of the archive itself is only limited by the
platform it is used on.
I think filenames should be specified case sensitive, but shouldn't match
each other case insitively. What I mean is that if the info file refers to
"Nemesis3.rom", the file must be named "Nemesis3.rom", not "nemesis3.rom" or
"NEMESIS3.ROM". In addition, if "Nemesis3.rom" exists, both "nemesis3.rom"
and "NEMESIS3.ROM" are no longer allowed filenames for different files
belonging to the same game. These rules will make sure there are no filename
problems on either case insensitive filesystems (Windows) nor case sensitive
ones (Unix).
I also think long filenames should be allowed. All fixed file names should
fit in 8.3, there is probably only one fixed file name which is the name of
the info file. All other file names are listed in the info file, so the
person who creates the archive can choose whether to use long filenames or
not. The only problem that arises from this decision is then a user of an 8.3
filesystem extracts the archive manually.
About special characters, I think it's safest not to allow any special
characters like letters with accents etc. It would be best not to allow
spaces either, those often cause a lot of trouble. Just use underscores.
3. Compression
TRD said storage capacity is cheap nowadays, so we don't have to use
compression. I don't agree, for a couple of reasons. For one thing, not every
file is a 16K ROM. MegaROMs are often 128K or 256K, some as large as 2MB. If
we include disk games as well, games of several megabytes exist. Having a
couple of hunderd games on your harddisk, the space does add up. And on a
real MSX storage size matters a lot more. The second reason is that network
capacity is not nearly as large as storage capacity, so we should also think
about download times and bandwidth use.
LZW is not a good choice: it is covered by a patent and the patent holder
(Unisys) won't allow compressors without a license. This caused the problems
with the GIF format.
ZIP-inflate and GZIP (is that the algorithm's name, if not, what is?) are
free and already implemented. Does anyone know what is the status of the LZH
algorithms ("-lh4-"/"-lh5-") and the PMA algorithm ("-pm2-")?
My concern is that some of the algorithms may be slow on real MSX, or require
much memory while unpacking. It would be nice if for example 32K ROM files
would run on 64K RAM machines. After the loader code and the system RAM, that
would leave about 16K of memory for the unpacking algorithm. Can anyone tell
us how much time and memory each compression algorithm takes?
Another thing I wonder is whether compressed info (title, company etc) and
pictures can be extracted fast enough. The typical use for this is to make an
emulator or front-end use this info to display a menu in which the user can
select the game to run. So it has to gather that info from an entire
directory in a short time.
4. Filename extension
I agree with Sean and Ricardo that .MSX is a good choice. There isn't any
"alive" format that uses that extension and it's easy to recognise the file
is an MSX game. On a case-sensitive filesystem, should it be upper case
(".MSX") or lower case (".msx")? I prefer lower case.
There may be a problem however with webservers trying to find a proper
mimetype for these files. It would be a safer choice to use for example
game.msx.zip if ZIP was chosen as an archive format. However, not all
platforms allow more than one extension. It seems that the platforms without
support for complex filenames are also the platforms that are rarely used in
combination with the internet, so maybe we should use a double extension
where possible and a single extension otherwise.
5. Info format
The info about the game can be stored in various ways:
- fixed-size fields
- variable-size fields
- ASCII
Fixed-size fields are rather limited. Variable-size fields are hard to edit.
ASCII is easy to edit, but harder to parse.
I think easy editing is an important property. Editing a complex format in a
hex editor is no fun, so separate editors would be needed. With an
ASCII-based format, any text editor suffices.
Inside an ASCII-based format there are several possible ways to format the
data. The most popular are INI and XML. XML is more powerful, but it has more
overhead than INI. Since I don't think we actually need the extra
expressiveness of XML, INI is probably the best choice. Again, I agree with
Ricardo.
6. Languages for info
It would be nice to allow different languages for the game info. Some games
have a Japanese title and it would be nice for Japanese users to read the
actual title and for non-Japanese users to read a translated title. And if
things like game tips are included in the info, multiple languages is really
useful.
Ricardo proposed this notation:
Name=(english: "Penguin Adventure",
nihongo: "yume tairiku adobentyaa")
There is some RFC or ISO standard that defines languages codes like "en_GB"
(British English), "nl" (Dutch) and "pt_BR" (Brazilian Portuguese). I think
it's a good idea to stick to that standard, so that we don't have to make a
list of language codes ourselves.
About the syntax, the .desktop file format used by KDE and GNOME stores it
like this:
Name[en]=Penguin Adventure
Name[jp]=yume tairiku adobentyaa
I think this notation is simpler than the one above.
7. Pictures
First of all, there is the question of including pictures or not. I like the
idea, although it does make the files larger. But if we're using a common
archive format (like ZIP, LZH or tar), anyone can remove the pictures if they
don't want them.
The next question is format. GIF is not a good choice because of the patent
issue. JPEG is free, but not very well suited for screen shots of MSX games.
JPEG is a good format for photos and similar images with a lot of fluent
colors (such as scans of the cartridge/disk/cover). MSX game screenshots have
only a few colors and very sharp lines, those pictures are better compressed
in a lossless format. PNG would be a good alternative for screenshots, it is
lossless and patent free and an implementation is available. PNG can handle
true color images, but it doesn't compress them as thightly as JPEG.
One option would be to allow multiple picture formats. The advantage is that
for every picture, the most suitable format can be chosen. The disadvantage
is that an emulator or front-end would either have to support many formats,
or be unable to display some images. On some systems it's possible to use
system libraries to handle images (KDE and Java do this: you simply provide a
filename and you'll get an image object, the system library handles the
actual format), but on more "bare" systems such as DOS, real MSX and embedded
systems (we should think of the future) this can be hard.
A very different approach to picture formats is to store the picture raw,
with a minimal header and no compression. If we choose to have compression on
the "outside", there is no point in compressing the picture files using a
lossless algorithm.
Another issue is the size (in pixels) of the picture. Is any size allowed, or
should the picture be of a fixed size? And if it is a fixed size, what would
that be. The maximum size an MSX screenshot can be is 512x448, which is
SCREEN6/7 in interlace and using overscan. The minimum size is 240x192, for
SCREEN0-WIDTH40. An compromise would be 256x212, this is good enough for 95%
of the games.
By the way, I'm not sure having scans etc in the game file is a good idea. It
increases the file size (a lot if it's a high-resolution scan) and someone
who wants to play the game probably looks at it once or twice and then never
uses it again. I think it would be better to use web sites for scans.
8. Hardware description
There are two aspects of hardware we should include:
- Location: internal or external?
Internal specifies what hardware there is inside a cartridge. External
specifies what kind of MSX machine (and with what extensions) should be used
to run this game.
- How hard is it needed?
Some hardware is mandatory, which means the game won't run without it. Other
hardware is optional. And there are different levels of optional, some
features really cripple the game when missing (for example SCC), other
features are just "nice to have" (for example more than 128K RAM on games
that do caching, like Pumpkin Adventure 3).
Take Nemesis 3 as an example:
- Konami5 mapper: internal mandatory
The game won't run without this.
- SCC: internal optional
The game will run without SCC, although suboptimal.
- MSX1: external mandatory
MSX1 is the minimal configuration.
- MSX2: external optional
If an MSX2 is found, a nicer palette is used.
- Game Master 2: external optional
Nemesis 3 can save in the GM2 SRAM.
Some hardware options that weren't mentioned already:
- minimum amount of main RAM
Japanese disk-based MSX2 games use 64K, but European games often use 128K.
- V9958
Although MSX2+ implies V9958, many games don't actually need the MSX2+
BIOS, only the VDP.
- MSX-MUSIC
This should be called MSX-MUSIC and not FM-PAC, because FM-PAC is the name
of a cartridge including MSX-MUSIC and an SRAM, many games use the
MSX-MUSIC but don't use the SRAM.
- copy protection?
DSK images don't include the information needed to emulate copy protection.
Maybe we should limit emulators to playing cracked games. But we could
also specify the type of copy protection and allow that to be emulated.
Copy protection isn't limited to DSK images, it could also be in a
cartridge or connected to a joystick port.
- tape
Metal Gear uses tape for its save games.
Apart from which aspects to describe, there is the question of how to
describe it. We discussed MegaROM mappers, they can be described either by
making a list of known mappers or by describing the mapper algorithm. The
advantage of known mappers is that it's simple and efficient. The advantage
of a mapper algorithm is that it supports any MegaROM mapper, even those that
aren't discovered yet. I'll do some research on mapper algorithms to see if
describing the algorithm is practical.
No matter what we decide on this issue, it would be a good idea to create a
list of all known mapper types. This includes an accurate description of how
the mapper works and a list of example games that use that mapper (testing
material). Sean Young gave a list of mappers that can be used as a base for
this list.
9. Save games
Many games allow saving. Save games can be stored in several ways:
- passwords
- tape
- game disk
- separate disk
- SRAM in cartridge
- PAC SRAM
- GM2 SRAM
- clock chip (alert time/date)
In the case of password saves the "hardware" is paper. But because papers get
lost easily, it may be nice to have the emulator remember the passwords. One
way is that the user types the password and the emulator stores it in an
ASCII file. An different approach is that the emulator takes a screen shot of
the screen displaying the password and saves that.
I think save games should be stored separately from the game itself. This
keeps the game file "untainted". And if the game file is compressed, changing
it would require a packer inside the emulator, instead of just an unpacker.
Although the PAC SRAM can be shared by many games, this is not a good idea.
The user must remember which blocks are in use by which games, and many games
don't even allow to user to choose the SRAM block but use a fixed block
instead. It's a lot easier to use a separate PAC SRAM for every game.
Should we standardise the actual save files, or only the way in which the
saving method is described? Standardising save files makes it possible to
exchange them between emulators, but it requires emulator authors to agree on
a single standard. I think we should try to standardise save files.
10. Web references
It would be nice to point a game player to sites containing more info about
this game. As I stated in point 7, I believe that scans belong on a web site,
not in the game file. Other nice info about games includes tips,
walkthroughs, game music, background info etc.
We could just put URLs in the game info file, but I don't think that's a good
idea. First problem is that an URL is often dependant on the host. If someone
changes their provider, the URL changes. So URLs change too often to store
them in the game file. Second problem is that new pages about a game may be
created and a new game file would have to be distributed to include that URL.
It doesn't make sense to download a game again just to have some extra URLs.
This can be solved by creating an indirection step. We could give every game
a unique code and then use some kind of internet service to retreive a list
of URLs for that code. A possible implementation is to give every game a
unique string and put a CGI somewhere that takes that ID string as a
parameter and displays an HTML page of links.
===
There are probably some issues I didn't address yet, but I've been typing for
several hours already so I'll stop now. I look forward to your reactions.
Bye,
Maarten
--
For info, see http://www.stack.nl/~wynke/MSX/listinfo.html
