Dan, IMHO, you're beating your head against the wall trying to avoid the obvious solution: don't use a large number of small records, use a small number of large records.
> Although I know there are opinions all over the map, there >has never been a concrete, universal reason not to use extremely small >records in PDBs until now. Taken literally, I'd agree that small records was not so much the problem as "large" record counts. IMHO, the definition of "large" in this context is around 8-10K records. There's also the issue of 20 bytes overhead per record (not 8) which on a percentage basis, may be significant compared to the data size or may even exceed it. But when you don't have very many records in the DB, it still doesn't equate to a big deal. So to me the issue has always been record counts, not record lengths. >Does this sound plausible? To me it sounds like starting down a slippery slope where you have to start doing device specific work, adequate testing becomes harder (ie, trying to ensure everything is still fine after heap compact operations etc), and you have no real recourse when it doesn't work right after a ROM upgrade (the T5's already had one). >But I've also only ever mucked >with the PDB format on disk, I don't know if it's different in Palm >storage, or different still in NAND flash. It has always been different on the device. The docs have always made this clear. Right at the start of the chapter on PDB and PRC database formats it says this: "NOTE: This chapter describes the format of PDB and PRC databases that are stored in files on desktop computers. When one of these databases is loaded into a Palm Powered handheld, the database is stored in memory in a format that is similar to, but different than the format described here. The in-memory format of PDBs and PRCs is subject to change and is not documented by PalmSource, Inc." It is also obvious that it is different by looking at the ratio of total size to data size on other devices beside the T5. On all the devices I've tried from OS 3.5 to OS5.x, the difference is 84 bytes base overhead, plus *20* bytes per record (not 8 as in the desktop format), plus appinfo or sortinfo blocks. But that 20 byte figure is subject to change, and I understand it has not always been the same size. At 30K records, that means you have about 600K of record header overhead which you could mostly eliminate, even on other devices. >If anybody has any greater knowledge of the internals here that would be >helpful, or knows how to squeeze more info out of PalmOne, or has a >different idea, I'd love to hear it! I don't know (or care) about the internals of how the OS stores it on the device. I'd much prefer to release applications which used officially supported database methods. Why invest time trying to circumvent what the T5 is doing? Just change how you store stuff. Since your data is read-only, it is really rather simple. One alternative is to switch to a stream file. I suspect the memory sector size is already a subset of the 4096 byte quasi "records", so you'd have zero bytes attributed to "slack". Then ahead of the data place a pseudo recordset header. Something like a 32-bit integer with the "record" count, followed by an array of 32-bit offsets to where each record begins in the stream file. Then all you need is a wrapper function to obtain a "record" from the stream file. It finds the offset to where the record would start, and the next one, using the difference to know the length of the record. There are many other ways to skin that cat too. I use a different technique than stream files, in part because I need to keep multiple "indexes" in various sort orders so I can quickly lookup or present data in a variety of sequences. But conceptually they all come down to having some type of jump table to tell you where to locate the data, and encapsulate the data retrieval behind wrapper functions so the main body of the program only deals with a logical record the size of what is currently one physical record for you. If your logical records are all a consistent size, then you don't even need the table of offsets -- you can simply compute a starting location within a stream file, or a record index and offset if you are packing multiple sets of data in longer records of a regular database. Even if your data is using variable length records now, remember that you'll be eliminating 20 bytes of header overhead per record by using packing. So you may be able to swtich to fixed sized logical "records" and still have the total size on the device be less than it takes on most devices now. And for sure you'd consume a fraction of the current size on the T5. IMHO, you'd be starting down a slippery slope to try to continue to use short records. I think you could implement a different storage paradigm with wrappers to return a given logical "record" in far less time than you'd spent trying to create an unsupported circumvention. As a side benefit, you well may take up less room even on other devices and find that you can (re)install the data much faster even though you aren't using Hot Sync. You'd have a fraction of the memory allocations which would need to be created and maintained. Plus you wouldn't have to worry that some ROM upgrade would break it all over again. Or some other new device, which may use a different sector size or not allow whatever circumventions you try on the T5. JMHO. YMMV. IANAL. Yada, yada. Doug -- For information on using the Palm Developer Forums, or to unsubscribe, please see http://www.palmos.com/dev/support/forums/
