--- [EMAIL PROTECTED] wrote:
> ----- Original Message ----
> > MemPage bitfield patch below.
> >
> > sizeof(MemPage) on Linux:
> >
> > original: 84
> > patched: 76
> > ...
> > Break-even for memory is 904/8 = 113 MemPage structs allocated.
>
> I didn't look at the code, so mind me :)
>
> If the MemPage are malloced individually (instead of being put in arrays),
> then they are 16 byte
> aligned on most platforms, making the allocated block effectively the same
> size (well, that
> depends on how many bytes are used by malloc before the user block in memory).
This patch does indeed save memory - on Linux at least.
Linux has a malloc mimimum resolution of 8 bytes, not 16.
The sqlite MemPage struct is appended to the end of each page, and the pages
are malloc'ed individually via:
pPg = sqliteMallocRaw( sizeof(*pPg) + pPager->pageSize
+ sizeof(u32) + pPager->nExtra
+ MEMDB*sizeof(PgHistory) );
Where nExtra is sizeof(MemPage) (more or less).
Linux's malloc() implementation, can support large non-power-of-2
malloc blocks, as this non-portable program demonstrates:
#include <malloc.h>
#include <stdlib.h>
#include <stdio.h>
void test(int page_size, int sizeof_MemPage) {
const int bytes_to_alloc = 48 + page_size + 4 + sizeof_MemPage;
printf("page_size = %d, bytes_to_alloc = %d\n",
page_size, bytes_to_alloc);
int i;
char* m1 = malloc(bytes_to_alloc);
for (i = 0; i < 5; ++i) {
char * m2 = malloc(bytes_to_alloc);
printf("0x%p: malloc resolution = %d bytes\n", m2, (int)(m2-m1));
m1 = m2;
}
}
main() {
int i=1024;
for (; i <= 8192; i*=2) {
test(i, 88); // test original struct size
test(i, 80); // test bitfield struct size
}
}
page_size = 1024, bytes_to_alloc = 1164
0x0x804a498: malloc resolution = 1168 bytes
0x0x804a928: malloc resolution = 1168 bytes
0x0x804adb8: malloc resolution = 1168 bytes
0x0x804b248: malloc resolution = 1168 bytes
0x0x804b6d8: malloc resolution = 1168 bytes
page_size = 1024, bytes_to_alloc = 1156
0x0x804bff0: malloc resolution = 1160 bytes
0x0x804c478: malloc resolution = 1160 bytes
0x0x804c900: malloc resolution = 1160 bytes
0x0x804cd88: malloc resolution = 1160 bytes
0x0x804d210: malloc resolution = 1160 bytes
page_size = 2048, bytes_to_alloc = 2188
0x0x804df28: malloc resolution = 2192 bytes
0x0x804e7b8: malloc resolution = 2192 bytes
0x0x804f048: malloc resolution = 2192 bytes
0x0x804f8d8: malloc resolution = 2192 bytes
0x0x8050168: malloc resolution = 2192 bytes
page_size = 2048, bytes_to_alloc = 2180
0x0x8051280: malloc resolution = 2184 bytes
0x0x8051b08: malloc resolution = 2184 bytes
0x0x8052390: malloc resolution = 2184 bytes
0x0x8052c18: malloc resolution = 2184 bytes
0x0x80534a0: malloc resolution = 2184 bytes
page_size = 4096, bytes_to_alloc = 4236
0x0x8054db8: malloc resolution = 4240 bytes
0x0x8055e48: malloc resolution = 4240 bytes
0x0x8056ed8: malloc resolution = 4240 bytes
0x0x8057f68: malloc resolution = 4240 bytes
0x0x8058ff8: malloc resolution = 4240 bytes
page_size = 4096, bytes_to_alloc = 4228
0x0x805b110: malloc resolution = 4232 bytes
0x0x805c198: malloc resolution = 4232 bytes
0x0x805d220: malloc resolution = 4232 bytes
0x0x805e2a8: malloc resolution = 4232 bytes
0x0x805f330: malloc resolution = 4232 bytes
page_size = 8192, bytes_to_alloc = 8332
0x0x8062448: malloc resolution = 8336 bytes
0x0x80644d8: malloc resolution = 8336 bytes
0x0x8066568: malloc resolution = 8336 bytes
0x0x80685f8: malloc resolution = 8336 bytes
0x0x806a688: malloc resolution = 8336 bytes
page_size = 8192, bytes_to_alloc = 8324
0x0x806e7a0: malloc resolution = 8328 bytes
0x0x8070828: malloc resolution = 8328 bytes
0x0x80728b0: malloc resolution = 8328 bytes
0x0x8074938: malloc resolution = 8328 bytes
0x0x80769c0: malloc resolution = 8328 bytes
Even though this sqlite page allocation scheme is fairly efficient
on Linux, it may be completely inefficient on platforms that have a
power-of-2 malloc allocator, such as some early BSD platforms.
On such a platform, a page_size of 8192 would result in
a malloc of 8332 bytes, which would have to be bumped up to
an effective 16384 bytes resulting in close to 50% memory
wastage.
I would recommend that SQLite not perform this clever merging
of page memory with PgHdr and MemPage because of its inefficiency
on platforms with sucky malloc implementations. These power-of-2
sqlite page_size'd raw page data should be individually seperately
allocated.
> Also, my take on bitfields is that they are not thread/multi processor
> friendly (there is no
> atomic "set bit"), and also compilers typically don't optimize well with that
> (so before
> applying this patch, I would test on other platforms than gcc linux x86).
Setting and reading individual bytes (u8 in sqlite-speak) are not
threadsafe either. Only reading/setting entire entire words
are threadsafe on most architectures.
It's moot anyway because generally one would not alter structs from
different threads without a mutex for synchronization.
____________________________________________________________________________________
Food fight? Enjoy some healthy debate
in the Yahoo! Answers Food & Drink Q&A.
http://answers.yahoo.com/dir/?link=list&sid=396545367
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
