>> You will have to maintain a list of free pages somewhere (when
>> an in-memory page in the middle of the database changes and moves
>> because it needs more on-disk pages to be allocated, you'll have
>> "free" pages in the middle of the on-disk file.)
>
> That the problem I'm facing right now. To store the page numbers,
> no problem. I can just say : the 4k page must compress to 2k - 8bytes
> and I have room to store the possible exception page number without
> polluting anything..
Yes.
> When a page is freed, however, it must be stored in a free list.
> I try to figure out how to do while keeping in mind that these pages
> are in a shared memory pool that can be used by many process and many
> threads in a process. The fact that getting a page will only occur in
> less than 0.1% may be an excuse to agressively lock the free list...
> I keep searching/reading code.
There's also the issue of recovery -- you have to be able to
rebuild that free list without reading through the whole
database file (which may be terabytes in size). So, there's
going to need to be a place where that freelist lives on disk.
>> Regardless, those numbers sound good. What was the range of
>> compression? How many compressed down to 1K?
>
> I didn't check that.. wait ... here : 16106 out of 230073 are <=
> 1024, 153076 out of 230073 are <= 1500 (2 are < 256 bytes :-).
OK, so it looks like 50% is what we're going to get, but that's
7% that are under 1K, which may make it reasonable to use 1K
fragments (on-disk chunks) instead of 2K fragments.
Usage: how about if we call in-memory pages pages, and on-disk
pages fragments? Saves on typing. :-)
Probably want to size fragments as a percentage of the page
size, either 25% or 50%.
>> We need to start thinking about recovery, too. If the system
>> crashes when you've only written one on-disk page of a two
>> on-disk page pair, how do you do recovery to guarantee that no
>> data is ever lost?
>
> Arg. Thanks about this one :-}
Right now, the mpool is blissfully ignorant of recovery issues,
it just has to write pages out. If the mpool starts modifying
pages, then those pages have to be recoverable. I *think* that
mpool won't have to care about the pages it compresses, all it
has to care about is the list of free pages.
--keith
------------------------------------
To unsubscribe from the htdig3-dev mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the SUBJECT of the message.
