Alan,
You bring up a good approach Alan. In fact, that was my initial
approach to the problem. However, as I found out, things REALLY get
ugly here.
Once I start allocating out of a pool of database records, a number of
things happen.
The first is that I have multiple pools to manage because a database
record can hold only <64K of data. To hold more than 64K of data, we
need multiple records.
The second thing is that now I have to write a memory allocator (which
is not bad if it's a fixed size allocator), take care of fragmentation
of the "heap"/compaction, etc. While I could do this, it would be
better to have the OS do it natively (using their APIs) because it's
more efficient. And, they are doing it anyway via their database API.
The third thing is that every write operation in the pool (compaction
especially) involves DmWrite operations. While they are relatively
fast, there would be a good number of them occurring on allocations and
deletes. While initially cheaper in terms of time to perform than a
DMNewRecord, as compaction occurs, that performance advantage
dissipates.
The fourth item to note is that multiple heaps don't work very well (as
Palm found out themselves in Palm OS 1.0/2.0). There can be issues
where I want to allocate a large block, but none of my pools can handle
it by itself. Collectively, they have enough space, but no single pool
can hold the entire record (Rhodes describes this really well in Palm
Programming, A Developers Guide). This means I either must fragment the
allocated memory block over multiple pools, rearrange the pool memory
across the different pools to find a good space, or disallow the
allocation. All of this boils down to a lot of overhead (even more than
DmNewRecord). Ultimately, the Palm OS 3.0 team didn't want to tackle
this issue because they have a single heap to allocate from. I don't
want to tackle this issue because the Palm people have already done that
work for me ;-)
Thanks Alan for the thoughts,
Mike
> -----Original Message-----
> From: Alan Pinstein [SMTP:[EMAIL PROTECTED]]
> Sent: Tuesday, March 02, 1999 6:23 PM
> To: [EMAIL PROTECTED]
> Subject: RE: More memory problems.....
>
> Mike-
>
> Would it be efficient to allocate pages in 'pools'? When the VMM
> starts up,
> it does a DmNewRecord() of a particular size, maybe 15k. Then you can
> lock
> that record and do your own chunk allocation/freeing out of this pool.
> Now
> you don't have a DmNewRecord() hit each time VMMHandleNew is called.
> You
> also save time by not having to lock/unlock the VM records as often,
> which
> while not as slow as DmNewRecord(), will save even more time. If your
> initial pool gets full, then you can allocate a new one.
>
> Just a thought...
>
> Alan Pinstein
> Synergy Solutions, Inc.
> http://www.synsolutions.com
> 1-800-210-5293
>
>
> >Ken,
> >
> >VMMHandleNew only allocates pages in storage RAM and sets up some
> >entries in some lookup tables.
> >
> >Ultimately, I almost always will still need to do a DmNewRecord which
> is
> >where the time killer is. It doesn't really matter whether it
> happens
> >in VMMHandleNew or VMMHandleLock, etc. You are correct in that I
> would
> >buy a performance benefit if the handle is never paged because the
> >record creation won't ever occur for it to matter in the timings.
> But,
> >as soon as the first page occurs, there's that hit to DmNewRecord.
> >
> >One thing that makes this whole project rather tricky is that even
> >internal lookup tables and such can get very large and may need to be
> >paged. Imagine allocating 10,000 100 byte blocks (about a megabyte
> in
> >storage space)--that lookup table isn't in dynamic RAM because it
> would
> >be something like 80K in size. So, there is an upper bound on the
> >amount of cached data and structures to manage it because it needs to
> be
> >stored somewhere, traversed quickly, yet be unpredictably large to
> >handle the most general circumstances.
> >
> >I have a list of efficiency measures on my todo list to which I'll
> add
> >your suggestion--I've held off making things "too" fast on them for
> now
> >for this big reason:
> >
> >It's easy to make working code fast, but it's much more difficult to
> >make fast code work 8-)
> >
> >Thanks for your input Ken.
> >
> >Mike
> >
> >> -----Original Message-----
> >> From: Kenneth Albanowski [SMTP:[EMAIL PROTECTED]]
> >> Sent: Tuesday, March 02, 1999 2:02 PM
> >> To: '[EMAIL PROTECTED]'
> >> Subject: RE: More memory problems.....
> >>
> >> On Tue, 2 Mar 1999, Mike Pellegrino wrote:
> >>
> >> > Neil,
> >> >
> >> > I have "finished" a beta virtual memory manager for the Palm
> which
> >> pages
> >> > dynamic RAM to the disk. What this means is that you can
> allocate
> >> more
> >> > RAM that physically available (I page it to storage RAM), but you
> >> can
> >> > only lock down handles if there is enough dynamic RAM to hold
> what's
> >> in
> >> > the handle. It may help you in your memory problems as far as
> >> storage
> >> > size goes.
> >> >
> >> > The docs aren't written yet (been a busy week so far-although the
> >> params
> >> > are identical to the Palm API calls) and I've not done enough
> >> stochastic
> >> > testing of the manager yet. But I'm happy to entertain beta
> users
> >> if
> >> > you're interested.
> >> >
> >> > The performance of VMMHandleNew sucks--it takes about 25-30 ms
> per
> >> > allocation. This is tied almost directly to DmNewRecord
> >> performance.
> >> > I'm currently investigating ways to improve this or amortize the
> >> cost
> >> > over a number of VMM operations. Other operations take much less
> >> time.
> >>
> >> Are you allocating underlying blocks in both the heap and storage
> >> during
> >> VMMHandleNew, or is the new handle just going into storage (since
> it
> >> starts out unlocked)? It sounds to me like it would probably
> (barring
> >> information to the contrary) make more sense to start out with an
> >> unlocked
> >> allocation on the heap, paging it out to storage only when heap
> fills
> >> up.
> >> Heap would then act as a cache for as many of the blocks as
> reasonable
> >> &
> >> possible, locked or unlocked, instead of only holding locked
> blocks.
> >>
> >> It would be more work to do it that way, though. :-)
> >>
> >> --
> >> Kenneth Albanowski ([EMAIL PROTECTED], CIS: 70705,126)
> >>
> >>
>
>
>
