Pack the data in whatever way makes sense, but he mentioned that he will only ever change a single byte per entry from true to false. If the 500,000 contact entries won't be changed, remove the flag byte from each record, and create a single large record as a bit field, either in the same or a different database. Change the corresponding bits in the bitfield, not the records themselves. That saves about 440,000 bytes in the database, and only requires a record of 62500 bytes to hold a 500,000 bit field. That way, hotsync only ever sees one dirty record.
Further, if the 500,000 contacts are human, they probably have a limited namespace. Tokenizing the name fields can probably reduce them to 2 bytes each. The other data items can be similarly compacted. -------------------- Jeff Loucks Work 425-284-1128 [EMAIL PROTECTED] Home 253-851-8908 [EMAIL PROTECTED] Mobile 253-691-8812 -----Original Message----- From: Douglas Handy [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 19, 2005 1:38 PM To: Palm Developer Forum Subject: Re: Advice on persistency please Logan / Jim / Clive, >>> I am developing an app. that will store approx. 500,000 contacts. >> ... Even on a 16M device, each record can average only about 32 bytes >If you could >get a 2-to-1 compression ratio through huffman coding, you'd be able >to store 15 characters of stuff per field plus null terminator in 16MB. I don't think any mention was made that it had to fit in 16MB devices. It obviously will be absolutely necessary to roll your own packing of "logical records" within physical PDB records since only a UInt16 can be used for the record index limiting you to 65535 records maximum prior to Cobalt. That said, this sounds like a good candidate to store on an external memory card and access via VFS, intead of internal storage heap. >> Syncing will likewise take forever. Not if you keep the physical record *count* down. >>Even if you pack many logical >> records in to one physical record, just transferring that much data is >> going to take ages. Not so. I do the same thing in some of my applications, and I can confirm the bottleneck is definately the record count, not the byte count. The largest example PDB which I could find for my apps is 6.39MB, but it syncs in just 15 seconds. This is more or less linear from the 2-3 seconds it takes me for a 1MB example file. I'd have to make something to generate random data since none of my real production files yield a 16+MB database, but based on all the tests I've done I'd fully expect a 16MB file to be able to sync in a minute or less, when keeping the PDB record count down. >Yes, there's probably no getting around that without some really >elaborate means like keeping a transaction log on both handheld and >desktop and then transferring the transaction logs through a custom >conduit during hotsync. Which would be messy! For read-only type data like this, my experience is that you're better off moving the entire data en masse. >So, I agree, it would be tough to do this on the Palm, but it's >not necessarily impossible. By far the biggest factor is keeping the physical record count down by using "logical records" within physical records. But to also keep the byte count down, I use a combination of "packed records" (ie only including fields with actual data, ala the AddressBook app) and compression on the text strings within a logical record. Then I insert multiple indices over the data so I can instantly display / search / sort the data using a variety of alternatives. The feasability of storing 500K contacts this way may hinge on how much data must be stored per contact. It may be more practical to store the data on a memory card and use VFS to access it, unless this is for a "single use" type corporate environment and it is known that enough internal memory will be available for the data. You can then use card readers to update the cards and instantly swap out the data available on a given device. And even the transfer to the card itself will be faster than you can achieve during a Hot Sync. But even using Hot Sync, you can transfer fairly large volumes of data relatively quickly provided you keep the record count down. YMMV. Doug -- For information on using the Palm Developer Forums, or to unsubscribe, please see http://www.palmos.com/dev/support/forums/
