Mike-
Great rebuttal...
>Ultimately, I almost always will still need to do a DmNewRecord which is
>where the time killer is. It doesn't really matter whether it happens
>in VMMHandleNew or VMMHandleLock, etc. You are correct in that I would
>buy a performance benefit if the handle is never paged because the
>record creation won't ever occur for it to matter in the timings. But,
>as soon as the first page occurs, there's that hit to DmNewRecord.
Got another idea. At boot time, why not allocate a block of 64 (or so)
1-byte length records? I don't know what the speed tradeoffs between
DmResizeRecord() and DmNewRecord() are, but it may be faster to resize a
record than to allocate a new one, since some of the overhead is already
taken care of... and I bet that DmResizeRecord() down to 1 byte when you're
DONE will be much faster than DmDeleteRecord(), but I could be wrong...
One other thing to consider (and I need someone from Palm to comment)... I
remember from DevCon that there was some discussion about Records vs.
Resources, and there were some speed differences in favor of resources for
certain things... they may be applicable here (that is, DmNewResource() may
be faster than DmNewRecord()).
Just a few more things to chew on!
Alan Pinstein
Synergy Solutions, Inc.
http://www.synsolutions.com
1-800-210-5293
>Alan,
>
>You bring up a good approach Alan. In fact, that was my initial
>approach to the problem. However, as I found out, things REALLY get
>ugly here.
>
>Once I start allocating out of a pool of database records, a number of
>things happen.
>
>The first is that I have multiple pools to manage because a database
>record can hold only <64K of data. To hold more than 64K of data, we
>need multiple records.
>
>The second thing is that now I have to write a memory allocator (which
>is not bad if it's a fixed size allocator), take care of fragmentation
>of the "heap"/compaction, etc. While I could do this, it would be
>better to have the OS do it natively (using their APIs) because it's
>more efficient. And, they are doing it anyway via their database API.
>
>The third thing is that every write operation in the pool (compaction
>especially) involves DmWrite operations. While they are relatively
>fast, there would be a good number of them occurring on allocations and
>deletes. While initially cheaper in terms of time to perform than a
>DMNewRecord, as compaction occurs, that performance advantage
>dissipates.
>
>The fourth item to note is that multiple heaps don't work very well (as
>Palm found out themselves in Palm OS 1.0/2.0). There can be issues
>where I want to allocate a large block, but none of my pools can handle
>it by itself. Collectively, they have enough space, but no single pool
>can hold the entire record (Rhodes describes this really well in Palm
>Programming, A Developers Guide). This means I either must fragment the
>allocated memory block over multiple pools, rearrange the pool memory
>across the different pools to find a good space, or disallow the
>allocation. All of this boils down to a lot of overhead (even more than
>DmNewRecord). Ultimately, the Palm OS 3.0 team didn't want to tackle
>this issue because they have a single heap to allocate from. I don't
>want to tackle this issue because the Palm people have already done that
>work for me ;-)
>
>Thanks Alan for the thoughts,
>
>Mike
>> -----Original Message-----
>> From: Alan Pinstein [SMTP:[EMAIL PROTECTED]]
>> Sent: Tuesday, March 02, 1999 6:23 PM
>> To: [EMAIL PROTECTED]
>> Subject: RE: More memory problems.....
>>
>> Mike-
>>
>> Would it be efficient to allocate pages in 'pools'? When the VMM
>> starts up,
>> it does a DmNewRecord() of a particular size, maybe 15k. Then you can
>> lock
>> that record and do your own chunk allocation/freeing out of this pool.
>> Now
>> you don't have a DmNewRecord() hit each time VMMHandleNew is called.
>> You
>> also save time by not having to lock/unlock the VM records as often,
>> which
>> while not as slow as DmNewRecord(), will save even more time. If your
>> initial pool gets full, then you can allocate a new one.
>>
>> Just a thought...
>>
>> Alan Pinstein
>> Synergy Solutions, Inc.
>> http://www.synsolutions.com
>> 1-800-210-5293
>>
>>
>> >Ken,
>> >
>> >VMMHandleNew only allocates pages in storage RAM and sets up some
>> >entries in some lookup tables.
>> >
>> >Ultimately, I almost always will still need to do a DmNewRecord which
>> is
>> >where the time killer is. It doesn't really matter whether it
>> happens
>> >in VMMHandleNew or VMMHandleLock, etc. You are correct in that I
>> would
>> >buy a performance benefit if the handle is never paged because the
>> >record creation won't ever occur for it to matter in the timings.
>> But,
>> >as soon as the first page occurs, there's that hit to DmNewRecord.
>> >
>> >One thing that makes this whole project rather tricky is that even
>> >internal lookup tables and such can get very large and may need to be
>> >paged. Imagine allocating 10,000 100 byte blocks (about a megabyte
>> in
>> >storage space)--that lookup table isn't in dynamic RAM because it
>> would
>> >be something like 80K in size. So, there is an upper bound on the
>> >amount of cached data and structures to manage it because it needs to
>> be
>> >stored somewhere, traversed quickly, yet be unpredictably large to
>> >handle the most general circumstances.
>> >
>> >I have a list of efficiency measures on my todo list to which I'll
>> add
>> >your suggestion--I've held off making things "too" fast on them for
>> now
>> >for this big reason:
>> >
>> >It's easy to make working code fast, but it's much more difficult to
>> >make fast code work 8-)
>> >
>> >Thanks for your input Ken.
>> >
>> >Mike
>> >
>> >> -----Original Message-----
>> >> From: Kenneth Albanowski [SMTP:[EMAIL PROTECTED]]
>> >> Sent: Tuesday, March 02, 1999 2:02 PM
>> >> To: '[EMAIL PROTECTED]'
>> >> Subject: RE: More memory problems.....
>> >>
>> >> On Tue, 2 Mar 1999, Mike Pellegrino wrote:
>> >>
>> >> > Neil,
>> >> >
>> >> > I have "finished" a beta virtual memory manager for the Palm
>> which
>> >> pages
>> >> > dynamic RAM to the disk. What this means is that you can
>> allocate
>> >> more
>> >> > RAM that physically available (I page it to storage RAM), but you
>> >> can
>> >> > only lock down handles if there is enough dynamic RAM to hold
>> what's
>> >> in
>> >> > the handle. It may help you in your memory problems as far as
>> >> storage
>> >> > size goes.
>> >> >
>> >> > The docs aren't written yet (been a busy week so far-although the
>> >> params
>> >> > are identical to the Palm API calls) and I've not done enough
>> >> stochastic
>> >> > testing of the manager yet. But I'm happy to entertain beta
>> users
>> >> if
>> >> > you're interested.
>> >> >
>> >> > The performance of VMMHandleNew sucks--it takes about 25-30 ms
>> per
>> >> > allocation. This is tied almost directly to DmNewRecord
>> >> performance.
>> >> > I'm currently investigating ways to improve this or amortize the
>> >> cost
>> >> > over a number of VMM operations. Other operations take much less
>> >> time.
>> >>
>> >> Are you allocating underlying blocks in both the heap and storage
>> >> during
>> >> VMMHandleNew, or is the new handle just going into storage (since
>> it
>> >> starts out unlocked)? It sounds to me like it would probably
>> (barring
>> >> information to the contrary) make more sense to start out with an
>> >> unlocked
>> >> allocation on the heap, paging it out to storage only when heap
>> fills
>> >> up.
>> >> Heap would then act as a cache for as many of the blocks as
>> reasonable
>> >> &
>> >> possible, locked or unlocked, instead of only holding locked
>> blocks.
>> >>
>> >> It would be more work to do it that way, though. :-)
>> >>
>> >> --
>> >> Kenneth Albanowski ([EMAIL PROTECTED], CIS: 70705,126)
>> >>
>> >>
>>
>>
>>
