Ralph Versteegen wrote:
On 12 April 2010 07:35, Mike Caron <[email protected]> wrote:
Ralph Versteegen wrote:
On 11 April 2010 13:49, <[email protected]> wrote:
pkmnfrk
2010-04-10 18:49:08 -0700 (Sat, 10 Apr 2010)
333
RELOAD now uses a hashtable to speed up working with the string table.
I am aware that util.bas has a hashtable implementation, but RELOAD's
uses its private heap, if applicable.
Also, added a MemoryUsage function, which tallies up all the memory used
on the aforementioned private heap. It's potentially very slow, so be
careful!
---
U wip/reload.bas
U wip/reload.bi
U wip/reloadtest.bas
Good god. THAT is NOT a hash function. You could have at least reused
the string hashing function from util.bas, strhash.
I actually forgot to check to see what yours was. I haven't had a chance to
profile this one.
While using a private heap is cool, I'm not convinced that it was
worth the effort. The effect will certainly be dwarfed by the speed
increases due to using zstrings, a hash table, and interning strings
in the string table instead of copying them. Have you timed the
difference? :)
I have not. The problem is that for documents that are large enough to reap
the benefits of having a private heap, RELOAD is all but unusable!
Which is strange, since I took precautions to the contrary...
You mean that it's unusably slow? Well I guess that explains why
you're spending so much time optimising, but how slow is unusable?
Well, one benchmark I use is to use xml2reload to transform a 64 meg XML
document into a RELOAD document. Loading the XML document (with libxml2)
takes ~10 seconds. Transforming that tree takes ~20 minutes, and
optimizing it takes a further 10 minutes. I forget how long it takes to
load the resulting document, but it's really stupid.
For small documents, it's pretty fast, but we would really need the
delay-loading to use it for anything larger (unless, of course, I figure
out where the bottle necks are!)
Because other than the difference memory allocation functions, I don't
think I see any reason for a new hashtable implementation.
In C++, we could template it, and pass in a memory allocating function.
Alas, poor -lang deprecated...
There's only a single callocate and deallocate there, wouldn't have
been hard to generalise it.
Anyway, the other reason I did it this way is because I don't like all that
byref stuff. I'd rather use pointers any day, since it makes it explicit
what is modifiable and what isn't.
I think I made 'this' byref because it was intented that the end user
would normally be using a HashTable, not a HashTable ptr. As for
hash_iter, looking back I might disagree with my use of byref there.
Also, I haven't picked through all this RELOAD stuff properly yet, but
now that you're using a hashtable do you still need Doc.strings[]?
Currently, I do, since the hashtable contains indicies into the string
table. I spit out the string table when I write the binary file, and read it
in again when I load it. Maybe I could do away with it, but it's not that
big a deal... sizeof(zstring ptr) * numStrings isn't that big a deal.
OK, better/easier to leave it then.
--
Mike
_______________________________________________
Ohrrpgce mailing list
[email protected]
http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
--
Mike
_______________________________________________
Ohrrpgce mailing list
[email protected]
http://lists.motherhamster.org/listinfo.cgi/ohrrpgce-motherhamster.org
